Systems and methods for rendering immersive environments

ABSTRACT

Disclosed herein are systems for rendering an immersive environment, the systems comprising at least one electronic device configured to be coupled to a body part of a user, the at least one electronic device comprising a sensor, an actuator, or both; a processor capable of being communicatively coupled to the at least one electronic device; and a rendering device capable of being communicatively coupled to the processor. The processor is configured to execute machine-executable instructions that, when executed by the processor, cause the processor to obtain data from or provide data to the at least one electronic device. The rendering device is configured to receive rendering information from the processor, and render the immersive environment based at least in part on the rendering information from the processor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/950,797, filed Nov. 17, 2020 and entitled SYSTEMS AND METHODS FORRENDERING IMMERSIVE ENVIRONMENTS” (Attorney Docket No. ARIS003U-C1),which is a continuation of U.S. patent application Ser. No. 16/081,685,filed Aug. 31, 2018 and entitled “SYSTEMS AND METHODS FOR RENDERINGIMMERSIVE ENVIRONMENTS” (Attorney Docket No. ARIS003U), which enteredthe national phase in the United States from PCT Application No.US2017/020213, filed Mar. 1, 2017, which claims priority to U.S.Provisional Patent Application No. 62/302,163, filed Mar. 1, 2016 andentitled “PATIENT-CARE PROCEDURE AND TRAINING USING SYNTHESIZEDIMAGERY.” All of the above-referenced applications are herebyincorporated by reference in their entireties for all purposes.

TECHNICAL FIELD

The present disclosure relates to systems and methods for rendering,recording, and using immersive environments in medical, gaming, andother fields.

BRIEF DESCRIPTION OF THE DRAWINGS

The various embodiments disclosed herein are illustrated by way ofexample, and not by way of limitation, in the figures of theaccompanying drawings and in which like reference numerals refer tosimilar elements and in which:

FIG. 1 is a block diagram of a system for tracking, in an immersiveenvironment, a positional sensor ingested by, injected into, or insertedinto a patient in accordance with some embodiments;

FIG. 2 is a block diagram of a system in which a coil unit isselectively tuned in frequency and power in accordance with someembodiments;

FIG. 3 is a block diagram of a system for auditory cardiographicanalysis in accordance with some embodiments;

FIG. 4 is a block diagram of a system in which a processor causes arotator to rotate an object in accordance with some embodiments;

FIG. 5 illustrates an exemplary surgical overlay system diagram;

FIG. 6 illustrates the surgical overlay program flow in accordance withsome embodiments;

FIG. 7 illustrates an exemplary laparoscopic system;

FIG. 8 illustrates the laparoscopic program flow in accordance with someembodiments;

FIG. 9 illustrates an exemplary head-up display (HUD) with two- andthree-dimensional elements in accordance with some embodiments;

FIG. 10 illustrates concurrent HUD display of an exemplary image andprocedural step in accordance with some embodiments;

FIG. 11 illustrates an example of a doctor using a scalpel with atracker and a monitor in accordance with some embodiments;

FIG. 12 illustrates an exemplary laparoscope path display and HUDcamera;

FIG. 13 illustrates exemplary HUD elements attached to a patient and anexemplary view presented in the HUD;

FIG. 14 illustrates an exemplary patient anatomy display with an elementof the anatomy virtually removed to permit visibility to otherwiseobstructed portions of the anatomy;

FIG. 15 illustrates an exemplary active dissection;

FIG. 16 illustrates an example of a virtual patient having an evidentsymptom (e.g., a rash or other topical ailment);

FIG. 17 illustrates an example of student learning in augmented reality;

FIG. 18 illustrates an example of a first-aid procedure;

FIG. 19 illustrates an exemplary IQ-test question presented in augmentedreality;

FIG. 20 illustrates an example of a virtual spider and vital-signmonitoring;

FIG. 21 illustrates an example of a potential phobia;

FIG. 22 illustrates an example of vital-sign elevation in the presenceof a particular visage;

FIG. 23 illustrates an exemplary pupil dilation and subsequent detectionthereof, including light variance and detection;

FIG. 24 illustrates an exemplary mirrored reflection of a patient anddoctor viewing sample plastic surgery outcomes (e.g., nose);

FIG. 25 illustrates exemplary control over an overlay of an anatomicalfeature (e.g., nose overlay controlled by surgeon) in real time;

FIG. 26 illustrates an example of a surgical procedure in progress withoverlay;

FIG. 27 illustrates an exemplary chest rise and detection/image capturethereof;

FIG. 28 illustrates an example of negative space exploration;

FIG. 29 illustrates an example of sequence matching based on featuresand/or perimeter;

FIG. 30 illustrates an exemplary frame-offset system;

FIG. 31 illustrates an exemplary circular test for two-dimensionalfeatures;

FIG. 32 illustrates an exemplary spherical test for three-dimensionalfeatures;

FIG. 33 illustrates a user's hands in different positions andcorresponding sensor data in accordance with some embodiments;

FIGS. 34A-34C illustrate exemplary systems that include at least oneelectronic device that is configured to be coupled to a body part of auser;

FIGS. 35A-35C illustrate exemplary systems that include a processor anda rendering device;

FIG. 36 illustrates an exemplary close-up view and surface view of asensor and interaction/recording therewith;

FIG. 37 illustrates an exemplary close-up view of an actuator, includingsurface emulation and playback;

FIG. 38 illustrates an exemplary analysis of a limp and differencebetween frames to enable detection of a fake limp;

FIGS. 39A and 39B illustrate selection between alternative images based,for example, on image blurriness; and

FIG. 40 illustrates an exemplary device for an augmented realitydisplay.

DETAILED DESCRIPTION

Methods, systems and system components are disclosed in variousembodiments for viewing and accurately locating patient organs, arteriesand other features prior to and during surgery, thereby reducingmorbidity and mortality due to surgical error associated with variancein patient feature location. In a number of embodiments, imaging data isinterpreted into an augmented reality (AR) or virtual reality (VR) viewof a patient, to be shown to a doctor, surgeon, or other medicalpractitioner during a procedure in order to enhance the accuracy andefficacy of the procedure. Methods and apparatuses interpret imagingdata into an AR or VR view of a subject for use by other usersincluding, but not limited to, insurance auditors, non-surgicalphysicians, nurses and legal professionals.

Methods and apparatuses for providing a heads-up display (HUD)displaying both AR path data and camera imagery for laparoscopic camerasduring medical procedures are also disclosed. In a number ofembodiments, locations of the laparoscope camera and/or carrier tube aretracked during laparoscopy, with transmission of the camera image to adevice, and overlay of the location and path data of the laparoscope inaugmented reality.

Various techniques and apparatuses for training and testing of surgicaland diagnostic skills using AR or VR and display of real patient datagathered by magnetic resonance imaging (MRI) are also disclosed. In anumber of embodiments, real patient data (e.g., composed from an MRI, CTscan, x-ray, or any other patient data source) is displayed to apractitioner/trainee and further enhanced through AR or VR to simulate avariety of conditions for testing and training.

An AR device is any device comprised of a computer controlled displaycapable of displaying either a transparent image atop real world data,such as glasses with an embedded transparent display mechanism, or adevice capable of displaying a composite image from a camera or otherimaging source coupled with overlaid three-dimensional data. A VR deviceis any device comprised of a computer-controlled display that covers theuser's vision and immerses the user in an immersive environment.

As used herein, the term “immersive environment” is a general termencompassing any or all of augmented-reality environments,virtual-reality environments, immersive-reality environments, andenhanced-reality environments.

I. Fluid Detection

Some embodiments disclosed herein relate to the detection of fluids. Forexample, some embodiments relate to a method and apparatus for sensingand displaying liposuction procedure data in virtual reality, augmentedreality, or other immersive environment. In liposuction, a cannula isused to break up or melt fat, after which the fat is suctioned up. Byadding a monitoring device to the cannula, the volume of materialsuctioned from the patient can be measured. This can be used to assistthe practitioner in ensuring that even amounts of material are removedfrom symmetrical areas, as well as to ensure that too much material isnot taken from the subject. The monitoring device consists of a digitalflow meter attached to a standard cannula. The flow meter is connectedto a display device to show the practitioner how much volume has flowedthrough the cannula. A button either on the cannula or device can beused to reset the flow meter.

Some embodiments relate to a method of sensing and displayingliposuction data, including but not limited to, volume of materialremoved from the patient, and mock-ups of post-surgical results.Cannulas for liposuction currently do not track the volume of materialremoved from a patient. By adding a flow meter to the cannula, anaccurate reading for how much fat has been removed from a patient can betaken. This allows a practitioner to ensure that they take a consistentamount of material from symmetrical parts of a client, and to ensurethat too much material is not taken, reducing potential harm to thesubject.

Some embodiments relate to a method and apparatus for detecting fluidusing a hygrometer attached to a cannula. As an example of a situationin which such an embodiment is useful, when draining fluid from behindthe eardrum, it is common practice for a doctor to punch a hole throughthe eardrum and add a stent. This method creates a permanent hole in themembrane of the eardrum, and exposes the patient to additional risk ofinfection. The repair of the hole is a complicated surgery.

In some embodiments, a small hygrometer is attached to the end of acannula, which can be used to detect fluid within internal cavities. Thereading from the hygrometer is transmitted to a display visible to thepractitioner. This display can be on a monitor such as an LCD or CRTmonitor, in an immersive environment, or any other display method thatis available to the practitioner.

To avoid creating a permanent hole in a patient's eardrum membrane, thedevice can be threaded through the Eustachian tube to the subject'saural cavity. If fluid is present, the hygrometer will indicate it tothe practitioner. Using this method, the practitioner can ensure thatall of the fluid is drained using the cannula to help ensure asuccessful procedure without the need to damage the membrane in thepatient's ear.

In some embodiments, a system for fluid detection comprises a cannula, amonitoring device (e.g., a flow meter, hygrometer, etc.) coupled to thecannula, and a display device coupled to the monitoring device. Thedisplay device presents an indication of the volume of material (e.g.,fluid) passing through the cannula. The monitoring device may be coupledto a button allowing a user to reset the monitoring device. For example,the button may be attached to the cannula, or it may be coupled to thedisplay device. The display device may include a screen, an LCD monitor,CRT monitor, an audio device, or any other mechanism to provideinformation to a user about the volume of material passing through thecannula. The display device may be in an immersive environment (e.g., itmay be visible in a heads-up display or visible/audible using anotherrendering device that presents an immersive environment).

II. Ingested-Positional-Sensor Tracking [AR/VR Assistance]

Some embodiments relate to a method and apparatus for tracking, invirtual reality, augmented reality, or other immersive environment, apositional sensor ingested by, inserted into, or injected into apatient. An ingested, injected, or inserted sensor can be used to trackthe digestive path of a subject to determine the path for an endoscope.The path can also be used to identify and locate blockages in thedigestive system.

Some embodiments relate to a method for tracking a positional sensoringested by, injected into, or inserted into a patient and tracked viaaugmented or virtual reality overlay. FIG. 1 illustrates a system 1200for tracking, in an immersive environment, a positional sensor ingestedby, injected into, or inserted into a patient. A sensor 1004 capable ofbroadcasting orientation, position, and/or speed data is encapsulated ina capsule. When a subject swallows the capsule, which is non-digestible,the capsule transmits data to a processor 1020 over a communication link1010C. The transmission can be made in radio frequency, Bluetooth,Wi-Fi, or any other method of wireless communication. The path data forthe capsule is taken in three dimensions, as well as the currentposition. The processor 1020 provides information to a rendering device1030 over the communication link 1010B. The provided information allowsthe rendering device to overlay data over the patient in an immersiveenvironment, allowing a practitioner to identify the path taken by thecapsule. The path of the capsule can be used, for example, to determinethe path for a gastroscopy procedure, or to identify blockages in thedigestive system. For example, a patient with a blockage in theintestine can swallow a small sensor (e.g., a macro-, micro-, ornano-sensor). The location and path of the sensor are tracked by thesensor 1004 and presented to the physician by the rendering device 1030in augmented reality, and the physician can look at the path of thesensor to determine at what point passage through the intestine stops.This helps the physician identify that a blockage exists, and to locatethe blockage.

In embodiments in which a sensor is injected into a patient, amacro-scale, micro-scale, or nano-scale sensor may be suspended in asolution for injection and tracked using receivers and software toaccurately track the location in three dimensions.

In some embodiments, a method for tracking a sensor within a patientcomprises receiving, from a positional sensor (e.g., a macro-, micro-,or nano-sensor) in the patient, a signal indicating the position and/orspeed of the positional sensor, and then, based on the signal,generating data representing the path of the positional sensor throughthe patient. A three-dimensional view of the patient overlaid by avirtual image of the path of the positional sensor through the patientmay then be rendered using the data. The positional sensor may beinjected into the patient, ingested by the patient, or inserted into thepatient. The signal from the positional sensor may be received over awireless channel or link (e.g., a radio-frequency, Wi-Fi, or Bluetoothlink).

In some embodiments, a system comprises a positional sensor (e.g., amacro-, micro-, or nano-sensor) that is configured to be ingested by,inserted in, or injected into a patient. If injected, the sensor may besuspended in a solution as described above. The positional sensorincludes a transmitter. The system also includes a receiver configuredto receive a signal (e.g., a wireless signal, such as radio-frequency,Wi-Fi, or Bluetooth) from the positional sensor, where the signalindicates a position and/or speed of the positional sensor within thepatient, and a processor coupled to the receiver. The processor obtainsthe signal (or information gleaned from the signal) from the receiverand, based thereon, executes computer instructions (e.g., a computerprogram) to determine a path of the positional sensor within thepatient. The system also includes a projector that is coupled to theprocessor. The projector obtains from the processor information that theprojector then uses to render a three-dimensional view of the patientoverlaid by a virtual image of the path of the positional sensor withinthe patient.

III. Adaptive Radiation Shielding, Including Dynamic Aperture Formation

Some embodiments relate to a method and an apparatus for adaptiveradiation shielding comprising a membrane or other container filled witha lead suspension solution. Additional membranes can be added to theapparatus containing solutions such as a ferromagnetic solution. Thelead suspension fluid shields the covered areas from radiation. Anaperture can be created by magnetizing the membrane and exposing it to areverse magnetic field. The reversed polarity will push away themagnetized particles, creating an aperture proportional in size to thestrength of the reversed field. The secondary membrane, such as aferromagnetic membrane, can be used for cooling to ensure that themembrane does not get too hot and damage the membrane.

Some embodiments relate to a method and apparatus for adaptive radiationshielding in radiation therapy. In radiation therapy, the non-targetedareas of a patient are covered using shielding to prevent damage tohealthy tissue. The targeted area, however, is not visible and the areasthat are covered or exposed are therefore determined using the bestjudgment of the person performing the procedure. Using augmentedreality, the location of the target area can be displayed on the patientin three-dimensional space, allowing for accurate placement of theradiation shielding. Additionally, using an adaptive radiation shield, acomputer can automatically determine and place the shielding without theneed for user interaction, allowing for a high degree of precision inthe placement of the shielding.

Some embodiments relate to a method and apparatus for adaptive radiationshielding for radiation therapy using augmented reality to direct thelocation and size of the exposure aperture. In some embodiments, asystem comprises an AR device, a camera or other imaging device, astandard radiation therapy setup, and optionally an audio capture devicefor recording and voice command input. An AR display of the radiationtarget is overlaid on the patient using methods described above. Usingthe visible target, the practitioner is able to accurately positionradiation shielding such that only the target area of the patient isexposed to radiation.

In an automatic embodiment, the radiation shielding is placed by acomputer using robotic manipulators or a shield that can be movedthrough automated means. A camera attached to the radiation sourcemonitors the path from the source to the target, and maneuvers theshielding into position. When the best location for the shielding,exposing the minimal non-target area possible, has been located, theshielding is fixed in place for the procedure.

In an interactive embodiment, the clinician is equipped with anaugmented-reality-enabled device allowing them to see radiation targetsin a patient in three-dimensional space. By aligning the adaptiveradiation shielding to cover all areas except the target area, theclinician can ensure that only the target area is hit by radiation, thusreducing the morbidity of adjacent areas.

In an automated embodiment, a computer is equipped with a camera toidentify the location of the patient, and is also able to control thelocation of the shielding. The computer then uses actuators to adjustthe size and position of the shielding to cover the non-targeted areas,allowing for a high degree of accuracy in radiation treatment.

For example, a patient being treated with radiation for breast cancerlies on the table used for radiation treatment. The practitioner,wearing a set of augmented reality glasses, is shown a visualization ofthe target tumor overlaid on the patient's body. Using thevisualization, the practitioner can accurately place the radiationshielding such that when viewed through the irradiating mechanism'scamera, only the tumor is visible. This reduces or eliminates damage tohealthy tissue during radiation therapy.

Some embodiments relate to an apparatus for adaptive radiation shieldingcomprising a membrane or other container filled with a lead suspensionsolution. Additional membranes can be added to the apparatus containingsolutions such as a ferromagnetic solution. The lead suspension fluidshields the covered areas from radiation. An aperture can be created bymagnetizing the membrane and exposing it to a reverse magnetic field.The reversed polarity will push away the magnetized particles, creatingan aperture proportional in size to the strength of the reversed field.Multiple magnetic fields can be used to shape the aperture. Thesecondary membrane, such as a ferromagnetic membrane, can be used forcooling to ensure that the membrane does not get too hot and damage themembrane. The magnets can be positioned either automatically or by apractitioner, or any combination in between.

For example, a patient being treated for an intestinal tumor lies on thetreatment table. The apparatus is placed over the patient's chest,abdomen, and thighs. Using an augmented reality overlay, a computer isable to visualize the tumor through a camera connected to theirradiating device. The apparatus is charged magnetically with apositive magnetic field. A magnet with a negative magnetic field isplaced over the site of the tumor, and the strength of the negativefield is adjusted by the computer to create an aperture large enough tosee the entire tumor. The tumor is then irradiated, with the healthytissue surrounding the tumor protected by the apparatus.

Some embodiments include an apparatus for shielding radiation in aradiation therapy procedure. For example, a mold can be made usingsilicone or another material to create a customized protective shieldusing any combination of lead, cadmium, indium, tin, antimony, cesium,barium, cerium, gadolinium, tungsten, bismuth, or other protectivematerial. This mold can also have an aperture sized and locatedspecifically for the target area.

In another example, radiation shielding is composed of many differentsegments, held to a supporting structure by Velcro or other adhesivemethod. The individual segments contain any combination of lead,cadmium, indium, tin, antimony, cesium, barium, cerium, gadolinium,tungsten, bismuth or other protective material. The segments can beadded or removed from the support structure to allow or block the flowof radiation to a given area. The support structure can be a rigidstructure designed to fit over a patient. The support structure can alsobe a flexible material that drapes over a patient. The support structurecan also be a garment to be worn by a patient.

In some embodiments, a system for use in radiation therapy of a patientincludes a radiation shield. The radiation shield may comprise a mold(e.g., silicone, lead, cadmium, indium, tin, antimony, cesium, barium,cerium, gadolinium, tungsten, bismuth, etc.). The radiation shield maycomprise a support structure and at least one segment (e.g., made oflead, cadmium, indium, tin, antimony, cesium, barium, cerium,gadolinium, tungsten, bismuth, etc.) coupled to the support structure,where the at least one segment may be permanent or removable from thesupport structure. The support structure may be rigid or flexible, or itmay be a garment to be worn by the patient.

The radiation shield comprises a membrane, which comprises a leadsuspension solution. The radiation shield is intended to be placed overat least a portion of the patient. The system also includes amagnetization system coupled to the radiation shield and configured tomagnetize the membrane, and expose the membrane to a reverse magneticfield to create an aperture in the radiation shield. The size of theaperture in the radiation shield may be dependent on the strength of thereverse magnetic field. The system also includes a rendering device(e.g., an augmented reality device) configured to render athree-dimensional virtual image of the aperture in the radiation shieldoverlaid on the patient. Optionally, the system may also include anaudio capture device configured to capture voice commands. Optionally,the system may also include a radiation therapy system configured toprovide the radiation therapy to the patient through the aperture.

The system may optionally also include at least one processor coupled tothe magnetization system and the rendering device, wherein the at leastone processor is configured to execute one or more instructions that,when executed, cause the at least one processor to obtain the size ofthe aperture (e.g., by calculating, retrieving, or receiving the size,either from a user or without user involvement), obtain a setting of themagnetization system (e.g., determine the setting without user input orbased on a user input) suitable to create an aperture of that size inthe radiation shield, cause the magnetization system to create theaperture of that size in the radiation shield, and cause the renderingdevice to render the three-dimensional virtual image of the aperture inthe radiation shield overlaid on the patient.

In some embodiments, the membrane is a first membrane, and the radiationshield further comprises a second membrane configured to cool the firstmembrane. In such embodiments, the second membrane may comprise aferromagnetic membrane.

The system may optionally also include means (e.g., a computer system)for positioning the radiation shield over the patient. For example, thesystem may include a camera coupled to a radiation delivery source,where the camera is configured to monitor a path from the radiationdelivery source to the patient, and a robotic manipulator coupled to thecamera and configured to place the radiation shield over the patient.

In some embodiments, a method of radiation therapy comprises placing aradiation shield over a patient, the radiation shield comprising amembrane, the membrane comprising a lead suspension solution,magnetizing the membrane, exposing the membrane to a reverse magneticfield to create an aperture in the radiation shield, through a renderingdevice, rendering a virtual image of the aperture overlaid on thepatient, using on the virtual image of the aperture overlaid on thepatient, positioning a radiation therapy system to deliver radiationtherapy to the patient through the aperture, and exposing the patient toradiation therapy through the aperture in the radiation shield.

IV. 3D Prosthetic Printing

Some embodiments relate to a method and apparatus for the creation andthree-dimensional (3D) printing of prosthetics. Using augmented orvirtual reality, a three-dimensional model for a prosthetic can becreated. This prosthetic model can then be exported in a format that canbe printed by three-dimensional printers.

Some embodiments relate to a method for creation and printing ofthree-dimensional models for prosthetics. Prosthetics can be created inan immersive environment through user interaction with gestures, voicecommands and other user input methods. A virtual subject is createdthrough common means of three-dimensional modeling, or by reconstructionfrom medical imaging. The subject can be a partial or complete entity,for example an entire person, or a portion of a person's anatomy. Theprosthetic is then created to match the shape and size required, andsaved to a storage medium. For example, a broken bone can often beshattered. By using a scanned image of the bone, a comparable bone (lefttibia vs. right tibia), or a generated shape, a three-dimensional modelcan be created. This model can be printed using three-dimensionalprinting methods, and a suitable replacement bone or bone segment can beused to repair the break.

The data saved to a storage medium can be used in three-dimensionalprinting in order to create a prosthetic from the virtual model. Forexample, a subject requiring a prosthetic foot is attended by apractitioner using a set of augmented reality glasses. The practitioneranalyzes the patient's existing foot, and selects the foot in augmentedreality. Using (for example) voice commands, the practitioner creates amirrored copy of the subject's foot. The foot is then overlaid in placeusing gestures to check the fit. The subject can also wear augmentedreality glasses to share in the immersive environment and see theprocess and fit for themselves. If the fit is not quite correct, thepractitioner may use a combination of voice and/or gesture controls toadjust the virtual foot until it appears correct. The practitioner canthen send the metrics, such as, but not limited to, shape and size to athree-dimensional printer for manufacture.

In some embodiments, a method of designing a prosthetic device for arecipient (e.g., a person, an animal, etc.) comprises presenting, in animmersive environment provided by a rendering device, a modelrepresenting at least a portion of the recipient, based on the modelrepresenting the at least a portion of the recipient, creating a modelof the prosthetic device in the immersive environment provided by therendering device, and storing information representing the model of theprosthetic device in a storage medium. In some embodiments, the methodfurther comprises obtaining a user input, and wherein presenting themodel comprises determining the model based at least in part on the userinput. Creating the model of the prosthetic device may compriseobtaining a user input (e.g., a gesture, voice command, keystroke, etc.)and creating the model based on that user input. Creating the model mayinvolve giving the prosthetic model a size or shape configured to fitthe at least a portion of the recipient. The prosthetic device may bemanufactured (e.g., using an additive manufacturing process such asthree-dimensional printing) based on the stored information representingthe model of the prosthetic device.

V. Multi-Coil/Customized-Coil Magnetic Resonance Imaging

Some embodiments include a method and apparatus for MRI using multipleradio-frequency (RF) coils. In a traditional MRI, there is a single RFcoil used to generate the excitation of targeted atoms. By usingmultiple RF coils instead of a single RF coil, operating independentlyor in synchronicity, a higher quality magnetic resonance image can berecorded.

Some embodiments relate to a method and apparatus for customization ofRF coils in MRI to create images with higher SNRs and better imagecontrast. RF coils in an MRI machine are fixed-position objects, eitheras part of the machine or as additional coils for specific sensing uses.When an MRI of the knee is being done, for example, the knee can befitted into a mold for the knee that contains an RF coil. In order toget a better SNR and image contrast, RF coils molded to the particularsubject's body (e.g., a custom-built, anatomically molded radiofrequency (RF) surface coil) can be used. These molded coils willprovide both more natural positioning of the patient and a better finalmagnetic resonance (MR) image.

Some embodiments relate to a method and apparatus for MRI comprising astandard MRI machine with the RF frequency coil replaced by multiplecoils operated independently or in a synchronized fashion in order togenerate an improved MR image.

Some embodiments include a method for using customized RF coils in MRimaging in order to create images with higher signal-to-noise ratio(SNR) and higher contrast. The RF coil used for excitation of atoms inan MR imaging sequence can be shaped to the subject area. Using a shapedRF coil (e.g., a custom-built, anatomically molded radio frequency (RF)surface coil) allows for a more accurate signal and better SNR.

In some embodiments, the shaped or customized RF coil can be selectivelytuned (e.g., in frequency and/or power) to allow for a clearer MR image.Selective tuning of available parameters (e.g., frequency, power, etc.)allows greater control over the image contrast and signal strength. FIG.2 illustrates a system 1300 in which a coil unit 1315, which comprisesone or more RF coils, is selectively tuned in frequency and power. Afrequency tuning unit 1305 controls the frequency provided to the coilunit 1315, and a power tuning unit 1310 controls the power provided tothe coil unit 1315. The frequency tuning unit 1305 and power tuning unit1310 may be coupled to a computer (not shown) that may be programmed toautomate the tuning process to adjust image quality. Alternatively, auser may manually adjust power and frequency.

In some embodiments, a magnetic resonance imaging system comprises afirst radio-frequency (RF) coil and a second RF coil, where one or bothof the first and second RF coils are customized. One or both of the RFcoils may be molded to a portion of a patient's body. A mold may includethe first and/or second RF coil.

In some embodiments, a method of performing magnetic resonance imaging(MRI) on a portion of a body of a patient comprises customizing (e.g.,shaping) a RF coil based on the portion of the patient's body andimaging the portion of the patient's body using the RF coil. The RF coilmay also be tuned (e.g., the applied power or frequency of the coil maybe adjusted either automatically, without human intervention, or inresponse to an input or instruction from a user). In some embodiments,the RF coil is included in a mold, and further comprising fitting theportion of the patient's body into the mold.

Imaging the portion of the patient's body using the RF coil may compriseobtaining a first MR image, obtaining a second MR image while or afterthe patient moves, and comparing the first and second MR images.

VI. User/Patient Activity Monitoring and Feedback

Some embodiments relate to a method and apparatus for analysis of MRimages taken while a patient is moving. By comparing different MR imagesfrom a moving patient, a practitioner can determine how the parts of aninjured limb or joint move. This detail allows for better diagnosis andtreatment of an injury. Moving MRI can be taken using a customized coilas discussed herein.

Some embodiments relate to a method for interacting with an immersiveenvironment using cerebral activity monitoring. In some embodiments,readings taken from a user's brain activity, by means such as, but notlimited to, alpha wave readings, beta wave readings, delta wavereadings, gamma wave readings, and theta wave readings,electroencephalography (EEG), magnetoencephalography (MEG), or cerebralimplant are used to control a user interface in an immersiveenvironment. Readings of frequency and amplitude may be used to controlelements of the user interface, either in concert or separately. Brainwaves may be used to measure and monitor changes in brain activity todetermine the efficacy of a treatment of neurological issues, epilepsy,etc.

In some embodiments, a method of monitoring user or patient movementcomprises at a first time, generating a first magnetic resonance (MR)image of a moving patient (e.g., using the magnetic resonance imagingsystem described above in the preceding section); at a second, latertime, generating a second MR image of the moving patient; and comparingthe second MR image to the first MR image.

In some embodiments, a method comprises obtaining a reading of a user'sbrain activity (e.g., an alpha, beta, delta, gamma, or theta wavereading taken using an EEG, MEG, or a cerebral implant), and using thereading, controlling, selecting, or modifying an element (e.g., a key, adisplay, an object, a brightness, etc.) of a user interface (e.g., avirtual keyboard, menu, peripheral, display, etc.) in an immersiveenvironment. The reading may be characterized by a frequency or anamplitude.

VII. Magneto-Stabilization of Patient Anatomy

Some embodiments relate to a method for magneto-stabilization of patientanatomy. A ferromagnetic fluid is injected into the area of a patientdesired to be stabilized. A magnetic device or material is then adheredor otherwise anchored to the patient in place at the location of thestabilization point. The magnetic field holds the ferromagnetic fluid inplace, stabilizing the targeted area. For example, a patient who has hada rhinoplasty could have the nasal area injected with ferromagneticfluid, and a magnetic bandage attached to the exterior of their nose. Asthe wound heals and swelling reduces, the interior anatomy is keptstabilized by the magnetic field.

In some embodiments, a method of magneto-stabilization of a patientcomprises injecting a ferromagnetic solution into an area of the patientto be stabilized and coupling a magnetic device or material to the area.

VIII. Tissue Separation

Some embodiments relate to a method for separating healthy tissue fromcancerous tissue. When surgery is performed to remove a cancerousgrowth, healthy tissue is removed with the cancerous tissue in order toensure that all of the cancerous tissue is removed. This procedure canresult in significantly more tissue being removed than is necessary forthe success of the surgery.

A ferrofluid or other magnetically responsive material is injected intothe area surrounding the cancerous tissue. The area is then exposed to astrong magnetic field, and the difference in absorption of the fluidbetween the healthy and cancerous tissue allows for magnetic separationof the two types of tissue.

IX. Identification of Microscopic Features Using High-Definition Camera

Some embodiments also relate to a method for identifying microscopicskin conditions using a high-definition camera, such as a stand-alonecamera or a camera integrated into or attached to a pair of glasses, aheadset, a helmet, or another wearable article. High-definition camerashave a much greater resolution than the human eye. By using ahigh-definition camera, a practitioner can identify microscopicorganisms and other such skin conditions in a subject eitherautomatically or with user interaction. In the case of automaticdetection, algorithms and pattern recognition are used to determinewhether microscopic organisms or skin conditions exist. In the case ofuser interaction, the user can optionally zoom in on an area to get amagnified view. With user interaction, automatic recognition can also beused to draw attention to details within the image and assist indiagnosis. Additionally, in some embodiments, using movement detection,practitioners are able to detect specific parasites such as lice ormites. For example, a patient with scabies (a mite that causes a rash inhumans) is viewed by a dermatologist. Using the enhanced high-definitioncamera, the dermatologist is able to see the mites, which wouldotherwise be invisible to the naked eye. This allows for certainty indiagnosing the patient with scabies and prescribing the appropriatemedication.

In some embodiments, a practitioner or technician places a sample on aslide, and an analysis system including a high-definition camera assistsin the analysis of a specimen. For example, the analysis system mayassist in the detection of a feature, a color, a movement, or any othercharacteristic. The analysis system assists in analysis andinterpretation of the data. In some embodiments, the analysis systemaccepts user inputs. In other embodiments, the analysis system performsits tasks automatically, without user input.

In some embodiments, a method of identifying a skin condition comprisesdirecting a high-definition camera toward an area of a patient's skinand identifying (e.g., using a computer to perform pattern recognition(either automated or with user input), detecting movement on thepatient's skin, etc.) the skin condition based on a view provided by thehigh-definition camera.

X. Cardiographic Analysis and Interpretation

Some embodiments relate to an apparatus for auditory cardiographicanalysis. Currently, physicians and other practitioners use astethoscope as the primary means of cardiographic auditory analysis. Astethoscope allows a practitioner to listen to the rhythm of a heart,however the analysis of the heartbeat is completely subjective, and thediagnostic outcome is entirely dependent on the skill of thepractitioner.

FIG. 3 illustrates a system 1400 for auditory cardiographic analysis. Insome embodiments, an apparatus for auditory cardiographic analysis (ACA)device 1040 comprises a sensor 1004, which is pressed against the chestof the subject in a location where the heartbeat can be heard.Optionally, an indicator 1042 on or of the ACA device 1040 informs theuser whether a sufficiently strong signal is present at the targetlocation. Alternatively or in addition, a stethoscope 1050 mayoptionally be coupled to the ACA device 1050 to allow the practitionerto listen and locate a strong signal. The signal from the heartbeat isthen digitized into a waveform and normalized, either by the ACA device1050 itself or, optionally, by a processor 1020 coupled to the ACAdevice 1040. The normalized data is then compared with a databank ofnormative heart rhythms, allowing for a rapid diagnosis of conditions.Optionally, the ACA device 1040 or the processor 1020 (if present)provides rendering instructions to a rendering device 1030, whichrenders information associated with the heartbeat signal, the compareddata, and/or the result of the comparison. The rendering device 1030 maybe, for example, a device that provides an immersive environment (asdiscussed elsewhere herein), or it may be a display (e.g., an LCDscreen) or other device that conveys information (e.g., a speaker, acomputer, a tablet, a mobile phone, etc.).

In another embodiment, a microphone (e.g., a parabolic microphone) isused instead of or in conjunction with the sensor 1004 on the subject'schest. The microphone is able to amplify and record sounds inaudible tothe human ear without amplification. The signal is then digitized into awaveform and normalized as explained above. In some embodiments, the ACAdevice includes a piezoelectric transducer that is capable of detectingthe sounds of a subject's heart.

In some embodiments, the ACA device 1040 comprises a receiver, which maybe, for example, a microphone. In such embodiments, the receiver may becoupled to the subject by a coupling fluid (e.g., water, ultrasound gel,etc.), or it may be suspended in a fluid or other medium, optionallysurrounded by a membrane that may additionally act as a coupling mediumor acoustic filter.

In some embodiments, physicians may be trained to listen for specificconditions using pre-recorded data and guided in learning with acomputer program assessing whether the physicians are correctlydiagnosing problems presented by the pre-recorded data. For example, thecomputer program may monitor a physician's performance in real time ornear real time as the physician is making diagnoses, and can providefeedback, hints, or help to ensure that a correct diagnosis is made andto improve the physician's training.

Some embodiments relate to an apparatus for rapid tracing andinterpretation of cardiographic rhythm anomalies. Cardiographic tracingand interpretation is currently done using multiple instruments, theecho-cardiogram (ECG) and stethoscope being the most commonly useddevices. The interpretation and analysis of the signals is donesubjectively by a practitioner, and the resultant outcome is dependenton the skill of the practitioner. Some embodiments include an extensionof the ACA device 1040 described above. As shown in FIG. 3, in someembodiments, in addition to the ACA device 1040, an ECG 1060 isoptionally added to the system 1400. The ECG 1060 is coupled to theprocessor 1020. The processor 1020 obtains data from the ECG 1060 inconcert with the data from the ACA device 1040 and compares thenormalized data from both the ECG 1060 and the sensor 1040 to a set ofnormative data. The data is fitted to the best match, and the result isreturned to the practitioner (e.g., through the rendering device 1030)for diagnostic purposes.

In some embodiments, a system for cardiographic analysis andinterpretation comprises a sensor configured to detect a heartbeat of apatient and a processor coupled to the sensor. The sensor may provide adigital signal representing the patient's heartbeat to the processor, orit may provide an analog signal to an analog-to-digital converter (ADC),which digitizes the signal before providing it to the processor. Theprocessor executes machine-executable instructions that cause theprocessor to normalize the digital signal, retrieve a reference signalfrom memory, and compare at least an aspect of the digital signal to atleast an aspect of the reference signal. The processor may also provideinformation indicating the result of the comparison (e.g., whether thenormalized digital signal comports with the reference signal in some way(amplitude, period, waveform shape or characteristics, etc.). The sensormay include an indicator (e.g., a light source, a display, a speaker,etc.) for indicating a level of the detected heartbeat of the patient.The sensor may include a microphone (e.g., a parabolic microphone). Thesystem may also include an electrocardiograph coupled to the processor,and the processor may be programmed to obtain a signal generated by theelectrocardiograph and compare an aspect of the signal generated by theelectrocardiograph (e.g., amplitude, period, waveform shape orcharacteristics, etc.) to that same aspect of the reference signal.

XI. Automated Diagnoses and Guided Treatment

Some embodiments relate to automated or user-guided devices and methodsfor performing diagnostic procedures, including, but not limited to,assessing, diagnosing and assisting in patient care in a triage or otheremergency setting. A practitioner using an augmented reality device cananalyze a patient to assess health based on signs and symptoms exhibitedby the patient. A diagnosis can be made using a databank of conditionsand symptoms. Directions can then be given by the device to assist thepractitioner in treatment of the patient.

As one example, in a triage situation a nurse wearing augmented realityglasses can connect a patient to vitals monitoring. The patient shows astachycardic. The nurse is able to contact a doctor using the augmentedreality glasses, and share with the doctor the vitals and view of thepatient. The doctor is then able to quickly assess whether the patientneeds immediate attention, and direct the nurse as to next steps.Alternately or in addition, video and vitals can be recorded by theaugmented reality glasses and transmitted to the doctor for review. Thedoctor can then contact the nurse, either via the glasses or throughother means, to indicate if escalation is required for a particularcase.

Some embodiments relate to an automated or user-guided device forperforming diagnostic procedures. This device can be, but is not limitedto, a probe, robot, automaton or other user or self-guided device. Usingartificial intelligence, the device is able to analyze symptoms andidentify conditions present in a subject. The subject symptoms areevaluated as well as other metrics, which may include, but are notlimited to, location, age, sex, environmental conditions, andnationality. For example, if a patient is thought to have a highlyinfectious disease, a robot is given direction to assess and analyze thepatient, performing a diagnosis. User input is given to direct the robotto look specifically for the suspected disease. This allows for correctdiagnosis of the patient without risking communication of the disease toa physician or other practitioner.

In some embodiments, a method for performing a diagnostic procedurecomprises viewing a patient using an augmented reality device (e.g., apair of glasses, a helmet, a headgear, etc.) and, using the augmentedreality device, sharing information (e.g., a vital sign, a video, etc.)about the patient with a remote practitioner. The method may alsoinclude receiving an instruction from the remote practitioner throughthe augmented reality device. The method may also include recording avideo of the patient through the augmented reality device and sharingthat video with the remote practitioner. The augmented reality devicemay be attached to or included as part of an automated or user-guideddevice, such as a probe, a robot, an automaton, etc. The automated oruser-guided device may be capable of analyzing a symptom exhibited bythe patient and identifying a condition based on the exhibited symptom.

XII. Audio Analysis, Translation, and Diagnostic Assistance

Some embodiments relate to a voice recognition system used to translatespeech between patients and practitioners in order to facilitatecommunication. Some embodiments relate to a system used to analyzespeech in a practitioner and patient environment to assist in diagnosisand verify plausibility of identified diagnoses. In some embodiments,the system is connected to a database of symptoms, diagnoses, andtreatment options. As the user and patient speak, their speech isanalyzed to identify symptoms and other relevant data. The data isprocessed, and a ranked or unranked list of possible diagnoses ispresented. The diagnoses also include treatment options, such asmedications and surgeries, for each particular diagnosis. When multiplediagnoses are possible, a list is provided to identify symptoms andsigns that could distinguish the conditions. The user can thenoptionally gather more information from the patient to refine thediagnoses. Some embodiments can also optionally provide information forreferrals to specialists.

In some embodiments, a method comprises capturing speech from a patient,extracting at least one characteristic from the captured speech,comparing the at least one characteristic from the captured speech to areference, and based on the comparison, providing at least one candidatediagnosis. For example, the at least one candidate diagnosis may includefirst and second candidate diagnoses that are provided in an order toindicate their respective likelihoods. The method may optionally alsoinclude providing at least one treatment option corresponding to the atleast one candidate diagnosis. The method may optionally also includeproviding additional information (e.g., an instruction, a referral to aspecialist, etc.) based on the at least one candidate diagnosis.

XIII. High Resolution Imaging Device

Some embodiments relate to a method and apparatus for capturing images.Images are captured using photodiodes coupled to an object (e.g., asphere, a cuboid, a half-dome, a strip, etc.). The object rotates at arapid rate, changing which photodiodes are able to capture an imagethrough the forward-facing aperture at any given time. The photodiodesare slightly offset in position surrounding the object (i.e., in theirplacements on the object). Due to the offset in position of thephotodiodes, and the speed of rotation, a very high-resolution image isable to be composited. A colour filter used to filter light into thephotodiodes rotates so that each photodiode alternates between thedifferent colours being filtered, such as red, green, and blue.Photodiodes used can be sensitive to any wavelength of electromagneticradiation, including but not limited to infrared, infrared, ultraviolet,and x-ray spectrums.

FIG. 4 illustrates a system 1600 in which a processor 1020 causes arotator 1090 to rotate an object as described above. The processor 1020is also coupled to the photodiodes 1080, which are coupled to at leastone filter 1085. The filter 1085 is a color filter used to filter lightinto the photodiodes 1080 as they rotate so that each photodiode 1080alternates between the different colors being filtered, such as red,green, and blue. The photodiodes 1080 can be sensitive to any wavelengthof electromagnetic radiation, including, but not limited to, infrared,infrared, ultraviolet, and x-ray spectrums.

In some embodiments, an imaging apparatus comprises an object (e.g., ageometric object such as a sphere, a half dome, a strip, etc.) that hasa plurality of photodiodes attached to its outer surface, means forrotating the object (e.g., a motor), and a processor for constructing animage based on signals generated by the plurality of photodiodes. Theimaging apparatus may also include at least one filter coupled to anddisposed between the photodiodes and the processor, the at least onefilter for filtering the signals generated by the photodiodes.

XIV. Visual AR/VR Medical Overlay Surgical Overlay

One embodiment relates to a method for displaying surgical targets andother pertinent medical and/or anatomical data in an augmented orvirtual reality surgical environment. When performing a surgery, thereexists a target location and/or anatomical part of the patient. Bydisplaying a three-dimensional rendered image, the efficacy of thesurgery can be increased, while reducing patient morbidity andmortality. The practitioner can optionally control the rendered image asdescribed below.

In augmented reality, the rendered image is seen by the user or users asa three-dimensional model of the patient's morphology overlaid atop thephysical patient. In the case of virtual reality, the patient morphologybecomes the three-dimensional model for the patient, and is treated asthe patient for the intended applications of some embodiments. In aprojection environment, the rendered image data is projected onto thesubject using a projector or projectors mounted with a view of thepatient. Multiple projectors are used to prevent the user or users frominterrupting the image, as well as to allow for a three-dimensionalimage to be presented.

The system is minimally comprised of a display device, the medicaloverlay software, patient morphology data, and a camera. In this minimalembodiment, the display device shows the image from the camera, and thesoftware interprets the image and places the patient morphological datain the correct location. Using the image from the camera, the softwareupdates the rendered image as described below.

In another embodiment [FIG. 5], the system is comprised of a pair ofaugmented reality glasses, tablet, display screen, virtual realityglasses or head-mounted display, sensors for tracking movement of the ARor VR device, the medical overlay software, a camera, an audio capturedevice, sensors for positional tracking of specific objects such asscalpels, hands or other instruments, optionally speakers, and/or a datastore for the patient morphology, which can be either pre-loaded ontothe device or transferred by network on demand. A projector can be usedin place of the AR or VR display. When the system is initialized [FIG.6, 101], the user first selects the procedure to be performed. The listof selectable procedures comes from a database of currently preparedpatient procedures. The data retrieved from the database is hereinreferenced as “procedural data,” which can include, but is not limitedto, the patient morphological data, patient information, proceduralinstructions, procedure time/date, and/or procedure location. Theprocedural data is then loaded from the database and stored in theprogram store being used for the execution of the system [FIG. 6, 102].This can be, but is not limited to, random access memory (RAM), a solidstate drive (SSD), a secure digital card (SD card), and/or a hard diskdrive (HDD) accessible to the system.

Optionally, the preferences of the current user or users are thenretrieved from a database of user preferences [FIG. 6, 103]. Thepreferences loaded are herein referred to as “practitioner preferences”and can include, but are not limited to, display brightness, HUDtransparency, HUD element location preferences, audio volume, and/orpreferred input method.

The system is then connected to sensor inputs to be monitored and/ortracked during execution. These sensors can be, but are not limited to,pulse monitors, blood pressure monitors, oxygen saturation monitors,and/or wireless sensor such as, but not limited to, RF positionalindicators. Sensor inputs are then verified to ensure that they arebeing correctly read [FIG. 6, 104]. The system displays to the user(s)the currently read value from each sensor in turn, and the user(s) thenconfirm that the value is correct. System execution is halted if theverification fails, unless user(s) specifically override theverification process. Following verification, visual targets are thenacquired by the system, the patient identity is confirmed, and therendered image position, orientation, and/or scale are verified [FIG. 6,105].

In order to visually track surgical instruments and other objects in theaugmented reality space, the user can hold the instrument in a locationvisible to the camera and request that the software identify theinstrument. Through user interaction it is determined whether thesoftware has correctly identified the implement. When the user issatisfied that the implement is being correctly identified, they thenindicate through a command—vocal or other user interface method—to trackthe identified instrument. Alternatively, or in addition, a trackingmarker can be attached to the instrument to be tracked and shown to thecamera, then indicated to the software through a user interface to begintracking the marker. Alternatively or additionally, one or more RF-basedtracking elements may be attached to or built into the instrument andengaged (e.g., Bluetooth pairing or other one-way or two-waycommunication link), at which point the software will begin tracking thetracking element(s).

Confirmation of the patient is done in two ways. Firstly, the patient'sinformation is encoded in the morphology data. The user compares theinformation in the morphology to the information associated with thepatient, whether on a hospital bracelet, clipboard, and/or otherstandard method of identifying patients. The morphology will also matchonly the correct patient, and therefore the rendered image will appearonly when the correct patient is visible to the system.

The rendered image as a whole is anchored to the location of thepatient. Herein, rendered image anchoring refers to positioning featuresof the rendered image, such as, but not limited to, detected featuresand/or perimeter location, and thus the rendered image as a whole suchthat the rendered image features are fixed in position relative to thepositioning features. Feature detection, perimeter detection, and/orpoint cloud mapping are used in conjunction to determine an accuratethree-dimensional location, scale and orientation of the patient. Thesemetrics are updated continuously as the program executes, to ensure thatthe rendered image is always anchored correctly. Markers can also beused, such as surgical tattoos or other visual markers, to ensure thecorrect anchoring of the morphological model.

Prior to commencing the procedure, the user or users do a walk around ofthe patient to ensure that the rendered image is properly sized andaligned to the patient. If the alignment is incorrect, the user(s) cancorrect the alignment using any method of user input available on thedevice.

The three-dimensional rendered image is rendered on the device, which inthe case of AR glasses may be a transparent screen embedded in theglasses themselves. In the case of virtual reality, the rendered imagemay be rendered on the non-transparent VR display. In the case of aprojection system, the rendered image may be projected onto the patientfrom any number of projectors mounted for that purpose. Multipleprojectors allow the projection to be unobstructed by movement of theuser or users.

During the procedure, the rendered image is continually updated todisplay the current morphology of the patient [FIG. 6, 106]. As asurgeon makes incisions and opens portions of anatomy, the renderedimage is updated in real time to present a progressively deeper view andrendered image with respect to the patient morphology. Thisdepth-tracking operation of the display can also be overridden by theuser or users using gestures, voice commands or any other form of userinput. The user(s) are also able to individually select and manipulateportions of the displayed morphology, such as removing an organ from themodel to view behind or below the organ or to view the organ fromvarious perspectives and proximities (orbiting, panning, zooming). Forexample, the user(s) can also rotate and reorient the portion that hasbeen removed to see it from different angles, as well as adjusting thedisplay depth to see inside the segment. All of these viewing controlsmay be effected through user input such as gestures, voice commands,swipes, taps, mouse motion, keyboard control, etc. The user(s) are alsoable to zoom in on the model in any portion, whether it be a portionthat has been removed from the primary morphology or a portion of theprimary morphology or all of the morphology itself

Relative movement between the patient and system user(s)—and thus actualor perceived movement of the markers used to anchor the renderedimage—may be detected in several ways [FIG. 6, 107]. One such method isthe frame offset method described below. Supplementary information isalso provided using the positional sensors in the augmented or virtualreality device (e.g., in the AR/VR goggles, display-shield or otherrendering device). In the case of a projection system, the projector isin a fixed position and therefore supplementary information isunavailable. As the user moves, his or her location in three-dimensionalspace is updated in the software, which in turn updates the visiblerendered image model or virtual model [FIG. 6, 108]. The model is alsoadjusted based on positional changes in the patient [FIG. 6, 109].Transformation of the location, orientation, and/or scale of themorphological data is done using quaternion and/or matrix operations totransform, translate, and/or rotate the points in the data set [FIG. 6,110]. As the patient moves, the morphological data is transformed tomatch the adjusted positions of the patient, as explained in an examplebelow.

The positions of any tracked objects are then determined inthree-dimensional space, and their locations for the purpose of therendering image are updated and stored [FIG. 6, 111]. User input, asdescribed above, is then processed [FIG. 6, 112]. Once input has beenprocessed and the rendered image has been updated, the view is rendered[FIG. 6, 113]. While using a surgical overlay, audio, and/or visual cuesare given to the surgeon if they are approaching an area that has eitherbeen noted as an area to avoid or use caution. For example, if a surgeonis performing surgery on the intestinal tract and the scalpel is gettingclose to the patient's bowel, a visual and/or auditory proximity warningmay be rendered to inform the surgeon that they have come too close. Thewarning could, for example, be a red area displayed in augmentedreality. A recorded warning or warning sound could also be played.

Anatomical Overlay

Another embodiment also relates to a method and apparatus for providingan anatomical display in virtual reality, augmented reality or otherimmersive environment. Anatomical diagrams, anatomical models, andcadaver dissection are the de facto standard for teaching anatomy tomedical students. By providing anatomical data in an immersiveenvironment, anatomy can be learned in three dimensions. This anatomicalmodel can also include notes to be displayed to the user or users. Themodel is not limited to humans, and can also be used for veterinarypurposes using anatomical models of animals and other living organisms.The model can also be interacted with by the user or users, allowing fordissection and manipulation of individual components of the model.Selection of specific parts of the model can be made by any method ofuser input, including but not limited to voice, gesture, and/or deviceinput. More details of a selected model can be made available to theuser(s) visually or aurally.

In augmented or virtual reality, three-dimensional anatomical models aredisplayed in a location where no actual model exists. In augmentedreality, the model can optionally be overlaid over a marker or otherpositional indicator, or even at a fixed location relative to the useror users that may contain physical objects. The model is presented inthree dimensions, and the display of the model can also be manipulatedas outlined below.

An anatomical model is displayed in augmented reality using a systemcomprising an augmented reality device such as a tablet, glasses,projector(s), or other display medium; a camera; sensors for trackingpositional movement of the camera and/or user(s); optionally speakersand/or an audio capture device for audio feedback and input,respectively; and a data store for the patient morphology, which can beeither pre-loaded onto the device or transferred by network on demand.

Annotations are also optionally displayed to the user or users, alongwith the ability to open detailed descriptions of individual anatomicalcomponents. While examining or dissecting the anatomical model, the useror users are able to manipulate anatomical components and move them awayfrom the main model, examining them in detail in three dimensions. Theuser or users are also able to zoom in on particular sections or on theentire model to have a closer look. The user or users are also able torotate and reorient the model, as well as individual sections of themodel.

Users are able to dissect the virtual anatomical model using user inputcontrols. The model can also be dissected using surgical instruments,either real or virtual. Virtual instruments are pre-created andinstantiated within the immersive environment using any common userinput method. Real instruments can be tracked and used as describedabove. As the user or users dissect the virtual model they see eachindividual component of anatomy, and are able to dissect the individualcomponents. Users are also able to reverse their actions using anymethod of user input to undo their actions sequentially. The model canalso be reset to the original position at any time using a commandissued by user input.

The user or users are able to move around the virtual model in threedimensions. The model is fixed to a point in three-dimensional space,selected when the model is first initialized. The model can be movedfrom this space with user interaction, but is otherwise anchored inplace. The location is determined using a combination of the frameoffset methodology described below, as well as positional informationgiven by the device and/or camera. In augmented reality, the user orusers are able to navigate around the model by moving their body inrelation to the virtual model. In virtual reality, the user or users areable to move through the immersive environment using commands issued byuser input, in conjunction with head tracking and any other availablepositional tracking information.

Laparoscopic Overlay

Another embodiment relates to a method and apparatus for providing avisual display of laparoscopic information in virtual reality, augmentedreality, or other immersive environment.

Laparoscopic procedures involve a surgical camera (laparoscope) andsurgical tools. By displaying radiological images overlaid over apatient in augmented or virtual reality, surgical targets, such ascancerous growths, can be more accurately targeted and located by apractitioner. The location of the laparoscope and surgical tools canalso be displayed. The historical location of the laparoscope andsurgical tools can also be shown as path data. A practitioner could alsotake notes, either vocally or using pre-determined commands, gestures orother pre-determined user interface options.

In a laparoscopic surgery, the surgeon is unable to see the actuallocation of the laparoscopic devices. The augmented reality devicedisplays the current location of the laparoscopic heads, the historicallocations (path) of the laparoscopic heads, and/or a HUD [see below]that displays the laparoscopic camera view. The device also displays(optionally) morphological data as explained above.

The laparoscopic overlay [FIG. 7] is comprised of a laparoscopicsurgical setup, augmented reality or virtual reality device (includingcamera and positional tracking), software, positional trackers,positional receivers and an interface between the receiver and augmentedreality device. The positional trackers are paired with the receiver(s),and attached to the ends of the laparoscopic instruments. The receiversare connected, preferably wirelessly, to the augmented reality device.The laparoscopic camera is connected (preferably wirelessly) to theaugmented reality device.

When the laparoscopic procedure has started, the system is engaged [FIG.8, 201]. The transmitters are then tested to verify that communicationsare correct between the transmitters, receivers and software [FIG. 8,202]. A rendering image is then displayed showing the initial positionsof the transmitters, as well as the initial camera view from thelaparoscope [FIG. 8, 203].

The positions of the laparoscopic heads are transmitted at regularintervals, as quickly as the slowest component in the system is able tohandle [FIG. 8, 204]. In order to maintain accurate and currentpositional locations for the trackers, the tracker and receiver operateat as rapid of a frequency as they are able. The augmented realitydevice then requests from the receiver an updated position as often asit is able to display it. Only the most recent positional data isreturned to the augmented reality device for display. The image from thelaparoscopic camera is also requested [FIG. 8, 205].

The rendering image is updated using the current and historicalpositions of the laparascope trackers, as well as the camera image [FIG.8, 206]. The current positions are displayed to the user or users inaugmented reality, as well as the historical positions. This allows theuser(s) to see both the current location and the track taken to arriveat the current location. The camera view is also displayed in a HUD (seebelow). This process repeats [FIG. 8, 202] until the procedure hasfinished.

For example, laser eye resurfacing is a process of improving a patient'svision by resurfacing the cornea of an eye to more accurately focuslight on the patient's retina.

Another embodiment is comprised of an augmented reality display, cameraor other imaging device, laser, and/or a cutting tool [laser,mechanical, etc.]. The surface of the cornea is scanned, and a model iscreated in AR. This AR model is used to assist in guiding the surgeonwhile using a laser to alter the surface of the cornea. The AR model isdisplayed either as an overlay over the actual cornea, or as a displayin a HUD (see below).

Real-Time/Heads-Up Display

During medical procedures, patient vital statistics, imaging, and otherpatient data are often required for consultation. A real-time updatingdisplay of the aforementioned data allows a practitioner to focus on thepatient or task at hand without having to consult devices or papersources to monitor or retrieve information. A range can also be set totrigger an alarm should a vital leave the acceptable range.

For example, a surgeon performing an appendectomy with a HUD could havea display of patient vital statistics shown in augmented reality,allowing the surgeon to focus on the surgical procedure without havingto look away in order to ensure that the patient's blood pressureremained stable.

The HUD is comprised of an AR device or other display medium and sourceinputs, such as vital signs monitors. The HUD is configuredautomatically, in advance, or by user interaction to select the type ofsource data to be displayed. The data is then displayed in a locationdetermined automatically, in advance, or by user interaction. Thetransparency (alpha channel value) of the HUD elements can also beadjusted to allow for better visibility of the HUD item or underlyingdetail.

Once the source inputs have been connected to the HUD, the values areread at regular intervals and the HUD elements are updated with the newvalues.

Another embodiment relates to a method and apparatus for displaying aHUD composed of two and/or three-dimensional images superimposed on theenvironment.

A HUD can be used for a large variety of purposes. In an immersiveenvironment, a HUD gives a viewer consistent information that remainsvisible regardless of the viewing context. This data can be configuredto show different information based on pre-set conditions, userpreferences, environmental factors, and/or contextual data.

For example, a doctor seeing patients could have a HUD displayingpatient information triggered by facial recognition of the patients.Additionally, for each patient, the doctor could configure which datawould be most valuable to see, and have that specific data displayed inthe HUD either for a single visit or on a long-term basis.

Various embodiments disclosed herein relate to a method for providing anaugmented or virtual reality surgical overlay, comprised of elementsincluding, but not limited to, HUD, medical imaging display, vitalstatistics display, patient information display, procedural informationand other data.

The HUD is created using two or three-dimensional images or models, withadaptive portions related to the data to be displayed. In the case ofvital statistics, the data is streamed from a medical device connectedto the subject. The data is then fed into the software where it'sinterpreted based on the information to be displayed, and displayed asappropriate. For a patient's 02 saturation, for example, the raw dataexpressed as a percentage can be converted to an integral percentagenumber for display in the HUD.

In another embodiment, the HUD can be replaced with another viewingmedium such as, but not limited to, an LCD or CRT screen. This view doesnot necessarily, but may, include an immersive environment.

FIG. 9 shows a sample HUD configuration. The four vital signs beingmonitored, temperature, oxygen saturation, pulse rate and blood pressureare shown in the top left, top right, bottom left, and bottom rightcorners respectively. These displays are transparent and are in fixedpositions such that as the user or users turn their heads, the vitalsigns remain in a constant position relative to the camera.

Similarly, medical images in formats recognized by the software,including, but not limited to, DICOM, JPEG, png, bitmap, raw, and othersimilar formats, can be overlaid as a part of the HUD to allow thepractitioner to see them in an immersive environment at all times.

Patient information and vital statistics can also be displayed in asimilar manner, having been loaded from a medical database or otherpre-existing source. Data can also be manually entered.

Procedural directions and information are also available frompre-created sources. These procedures and methods can be stepped throughusing various forms of user interaction such as voice control, gesturecontrol or other control method.

FIG. 10 shows a HUD identical to FIG. 9, however on the left below thetemperature stats a guide can be shown giving instructions to the useron how to perform a procedure. As each step is completed, the guide isupdated either automatically or with user interaction.

Another embodiment relates to a method for displaying surgical targetsand other pertinent medical and/or anatomical data in an augmented orvirtual reality environment.

The target area can be selected through a three-dimensional immersiveenvironment. Target areas can also be selected by a practitioner on apatient using an overlay. Target areas can also be selected using a pen,finger or other positional device. The targets can also be displayed ona conventional display, such as but not limited to, an LCD or CRTscreen. Positional tracking information sent from a surgical implementor other tracking method can be used to identify to the practitionerwhere the implement or tracker is relative to the targeted location onthe CRT screen.

FIG. 11 shows a scalpel equipped with a positional tracker (left) beingused by a surgeon. On the right, a display device is shown with a mockup of a patient's morphology. The X on the display device represents thecurrent location of the scalpel, while the circle represents thesurgical target location. By looking at the display device, the surgeoncan determine when they've reached the correct location to begin theirincision.

For example, a surgeon reviews an MR image of a patient's abdomen. Thetarget location for an abdominal surgery is identified from the image.Using a diagram of the patient, the surgeon marks the target area.During surgery, the diagram is displayed on a monitor next to thepatient. A positional tracker attached to a scalpel displays theposition of the scalpel relative to the patient on the monitor as well.When the position of the scalpel matches the position of the target, thesurgeon is able to see on the monitor that the positions are the same.This indicates to the surgeon that the right location has been found tobegin the surgery.

In another example, a surgeon performing surgery to remove a tumor on apatient's heart can separate the patient's heart from the body inaugmented reality, move the heart away from the patient, and inspect theheart and associated tumor in three-dimensional space. This allows thesurgeon to better assess the location of the tumor, as well as to planthe best route of access for its removal. This will allow for moresurgical accuracy tailored to individuals. This view can also be sharedvia network with other users for consultation or other uses.

In another example, an instructor uses a positional tracker attached toa pen or other implement to test students' knowledge. The instructor haspreviously identified a target for a surgical procedure, and thestudents are asked to locate the target using the implement. Theinstructor, wearing a pair of augmented reality glasses, can view theproximity of the students' answer to the actual target. In anotherversion of this example, the student could be shown a radiological imageand asked to identify the correct target location from the image.

In another example, a physiotherapist uses morphological images todisplay a spinal injury. Using this overlay, the physiotherapist is ableto accurately assist the patient without causing further injury ordamage to the spine.

In another example, a patient bends their right arm during a procedurefor which a rendered image is used. The morphological source data isthen updated to reflect the new position of the bent arm. The cameraimage is analyzed to determine the direction and degree of the bend inthe arm at various points. Using this direction and degree, themorphological data is updated to reflect new positions for each pointthat has moved using standard quaternion and/or matrix basedtransformation methods.

Another embodiment relates to a method for providing an augmented orvirtual reality surgical overlay for laparoscopic procedures, comprisedof elements including, but not limited to, mapping of laparoscopicdevice path, display of laparoscopic device position, display oflaparoscopic imaging data, and/or system for taking notes generally andrelated to specific points.

Laparoscopes are currently equipped with a camera for viewing the insideof a patient or other area in order to perform surgery non-invasively.By mounting a transmitter on the end of the laparoscope, and used inconjunction with a receiver connected to software, the location andhistorical path of the laparoscope can be tracked and displayed in animmersive environment. The transmitter can be using any frequencyallowable within a surgical environment, such as, but not limited to,RF, Bluetooth, or Wi-Fi.

The data from the camera can also be read and displayed in real time inan immersive environment, either as a primary display or a HUD. Having adisplay in view during the entire procedure allows for reduced morbidityand mortality during the procedure.

FIG. 12 shows a laparoscopic procedure in progress. On the left theaugmented reality paths and tips of the laparoscopic instruments can beseen. On the right the camera view from the laparoscope is shown, whichwould be visible in the HUD of the surgeon or other user.

Additionally, the practitioner can make notes using a user interfacecomprised of voice recognition, gesture recognition, and/or other formsof inputs. A practitioner can use a predetermined gesture to identifythe location where they would like to annotate. Once the gesture hasbeen recognized, they can then speak the note they wish to take, whichwill be interpreted by well-known methods of voice recognition andconverted to text to be displayed in the HUD. These notes are alsorecorded for future reference.

For example, when planning for an appendectomy, a surgeon reviews thepatient's model. While inspecting the model and planning a route for thesurgery, the surgeon notices that the patient has a postilieal appendix.Due to the position of the appendix, the surgeon makes a note on themodel to be cautious of the ileum, with the hope of reducing the risk ofaccidental damage to the ileum.

For example, in laparoscopic cholecystectomy (surgical removal of thegall bladder), a laparoscope is used to locate the gall bladder forremoval. The display from the laparoscope is traditionally shown on ascreen next to the surgical area, and the surgeon is unable to see thelaparoscope's location or path. Further, the surgeon is unable to focuson the laparoscope output while looking at the patient. Using augmentedreality, the laparoscope position and its path through the patient'sbody can be displayed directly on the patient's body. The camera viewfrom the laparoscope can also be shown in the HUD, allowing the surgeonto see both the patient and the camera simultaneously.

Another embodiment relates to a method for displaying a HUD in augmentedor virtual reality composed of two or three-dimensional imagessuperimposed on or integrated into the environment being viewed.

A HUD is used to display data to a user in an immersive environment. Theelements of the HUD can be either fixed positionally to the view of theuser, to locations in the real or immersive environment, or acombination of both. For example, in displaying patient data to a user,some elements of the HUD could be fixed to the location of the patient(such as heart rate, blood pressure), while other elements could befixed to the view of the practitioner, such as radiological images,patient information, or procedural notes.

FIG. 13 shows two separate HUD elements. One, a pulse rate for thepatient, is anchored to the patient's location and remains in the sameplace in three-dimensional space as the user moves about. The second,which includes the patient's name, age and blood type, is fixed to thebottom left corner of the view.

For example, a doctor doing rounds between hospital rooms can have a HUDfor display of patient vital signs. As the doctor passes from onepatient room to another, the HUD updates with the patient the doctor iscurrently visiting.

In another example, during anaesthesia, a patient must be monitoredconstantly to ensure that their vital signs remain stable and in anacceptable range. Using an augmented reality device connected to vitalsign monitors, the person monitoring the patient can keep the vitalsigns in view at all times using an augmented HUD. This allows themonitor to perform other tasks while continuing to monitor the patientunder anesthetic. Multiple patient vital signs can also be connected toa single augmented reality device, allowing a single monitor to watchover multiple patients under anesthetic.

In another embodiment, first responders (e.g., EMT) can use an immersiveenvironment device programmed with early life saving processes. Apatient's vitals can be streamed to the device, and based on symptoms aprotocol is initiated to provide step by step life saving steps to thefirst responder.

In another embodiment, a nurse or resident on call has an immersiveenvironment device connected to patients' emergency buttons. When theemergency button is pressed, the patient's vitals and location areconnected to the device. The device can also be connected to thephysician in charge of the patient, who may be present in hospital or oncall. The nurse, resident or physician can then communicate with thepatient and each other to determine the correct steps to ensure thesafety of the patient.

For example, a nurse watching a ward floor at night is at a desk outsidethe patients' rooms. A HUD displayed in augmented reality is shown tothe nurse while filling out paperwork. A patient presses the emergencybutton. The vitals for the patient are immediately displayed in the HUD,and the nurse sees that the patient is tachycardic. The patient historyin the HUD shows no history of tachycardia or related conditions, so thenurse initiates a call to the doctor on call through the augmentedreality device. The doctor, who is at home, is able to view thesituation through the camera on the nurse's augmented reality device andwalk the nurse through the steps of stabilizing the patient whiletravelling to the hospital.

Another embodiment relates to a method for using augmented reality inlaser eye surgery.

By using an augmented reality overlay in a laser eye surgery procedures,better accuracy can be given to the surgeon. The eye can be scanned andthe surgical target overlaid over the eye. This target can also bemanipulated as described below, including the ability to move it toanother location, zoom, rotate, and otherwise manipulate it for closerinspection and note taking.

For example, the cornea of a user can be scanned by high definitioncamera or other means in a LASIK surgery. The desired shape of thecornea is compared to the scanned cornea's surface. An augmented realityoverlay of the differences is shown on the cornea of the subject duringresurfacing, with the virtual object being updated as the surgeonreshapes the cornea. This allows the surgeon to be certain of correctlyresurfacing all portions of the cornea during the procedure, reducingsurgical error.

In another example, the back of a patient's eye is scanned and mapped tofind a tear in the retina. A small tear is located and processed into anaugmented reality morphology model. This morphological model issuperimposed over the patient's eye, showing the practitioner accuratelythe location of the retinal tear. The tear can then be repaired easilyand safely using an argon laser.

Another embodiment relates to a method for analyzing radiological imageswith a moving patient for diagnostic purposes.

Using radiological images taken at different points of motion can showchanges in joint position and possible fluid buildups, for example. Thiscan also be used to diagnose conditions such as arthritis.

XV. AR/VR-Assisted Medical Training/Learning/Simulation/Testing

Another embodiment relates to a method for combining gross anatomy withproblem based learning (PBL).

Gross anatomy and PBL are two different methods used in the teaching ofanatomy. By combining both methods, an enhanced understanding can be hadby the student.

Another embodiment relates to a method and apparatus for providingmedical simulations in virtual reality, augmented reality, or otherimmersive environment.

Medical and diagnostic training is primarily provided through classroomlearning, followed by a period of residency where a student learns byseeing real patients. The ability to train in surgical and diagnosticprocedures, however, is currently lacking. Using simulations in animmersive environment, a student can receive hands-on practice withoutrisk to patients, and with the ability for an instructor or peer tomonitor, grade and assist. Group simulations can also be done, allowingmultiple students and/or instructors to view and perform in concert.These simulations can also be used for examination of students in orderto determine suitability for practice in the field.

Often in practice, surgeons do not use the most up-to-date methods.Surgical knowledge is typically passed on through schooling andresidency. When a surgeon is taught how to perform a particularprocedure, they will learn the method used by the instructor. Theinstructor in turn will be teaching the method they are most familiarwith, which may not be a current method. Using augmented or virtualreality as a training mechanism, practitioners can be kept up to datewith the latest techniques in performing procedures. Through interactivesimulations, a surgeon can train in the most current methods ofperforming a particular procedure. Software can also be updatedregularly to ensure that the most up-to-date methods are available fortraining, reducing morbidity and mortality in patients.

Another embodiment relates to a method and apparatus for teaching andtesting using artificial intelligence coupled with virtual and/oraugmented reality.

Using an immersive environment to visually present materials to astudent, artificial intelligence algorithms can be applied in order totest whether the material has been learned by the user, and to adjustthe rate and style of teaching to match the needs and preferences of theuser.

Another embodiment relates to a method and apparatus for using recordedsensor data for training. In some embodiments, recorded sensor data isused in simulations to train users in diagnostic medicine. Sensor datais recorded for specific diagnostic procedures, such as a prostate exam,and replayed during a simulated diagnostic procedure to teach the useror users what a healthy or unhealthy diagnosis feels like. For example,sensor data may be recorded for the feeling of a normal vs. enlargedspleen in the detection of mononucleosis. The recorded data may then beplayed back during training to allow students to feel the differencebetween the two states, allowing for better diagnosis in practice. Insome embodiments, the recorded data is also compared to new dataacquired by sensors, allowing for a diagnosis using the device.

Another embodiment relates to a method and apparatus for first aidtraining using augmented reality, virtual reality, or another immersiveenvironment.

First aid training is a common form of medical training available to alarge portion of the population. Traditional first aid training,however, doesn't allow the user or users to experience real situationsin which first aid could be necessary. By using an immersiveenvironment, first aid situations can be simulated, and the user(s) canbe given guidance and training in the necessary steps to perform therequired aid. The simulation can also be used to evaluate theperformance of the user(s) and determine whether they should be deemedcompetent in taking action in a first aid situation.

Another embodiment includes a method and apparatus for intelligencequotient (IQ) testing using augmented reality, virtual reality, or otherimmersive environment.

IQ testing is done using a variety of tests involving different aspectsof intelligence. These tests can be administered in an immersiveenvironment, and the results evaluated automatically, or with any degreeof evaluator interaction. Normally an examiner monitors the subjectduring the test to evaluate performance. This is frequently a cause ofanxiety for the subject being tested, which can lead to less thanoptimal performance. Using an immersive environment test removes theneed for an examiner to monitor the subject.

Another embodiment is a method for teaching students using augmented orvirtual reality combined with artificial intelligence.

Another embodiment is a game in which the user or users are instructedwhich simulated organ to remove from a virtual patient. If the usersuccessfully removes the organ, they receive a point. If they do not,they are rewarded with a sound or other feedback mechanism. Turns aretaken by multiple users to reach the highest score and determine awinner.

Another embodiment relates to a method for providing an augmented orvirtual reality anatomical display, comprised of elements including, butnot limited to, anatomical diagramming and labelling, veterinaryanatomy, and dissection simulations.

Anatomical display can be done in augmented or virtual reality usingpre-created and optionally annotated models. These models are displayedin three dimensions, and can be interacted with by the user or users. Byusing voice, gesture and other user controls, the user or users canmanipulate individual parts of the body. The user(s) can also specifywhich layers and portions of the anatomy to be displayed. Individualparts, for example organs, can be separated from the main model forcloser inspection and to provide greater detail about the selectedfeature.

FIG. 14 shows an augmented reality anatomical model with a kidneyremoved for closer inspection. The kidney has been modified to display across section.

These diagrams can be of humans, animals or any living organism.Simulations can also be prepared for dissection, allowing a student tointeract using a controller, gestures, or other means of user interfacein order to attempt to perform a dissection, with feedback given to tellthe user if they've made a mistake.

FIG. 15 shows a user dissecting a virtual cadaver, removing a section ofthe epidermis to reveal the tissue underneath.

For example, in a classroom environment, this cuts out the need forgross anatomy, which has fallen out of favour due to health regulations.Instructors and students can explore anatomy in a virtual body, ratherthan having to deal with the costs and regulatory issues surrounding theuse of cadavers, and in a more hands-on fashion than that afforded bytraditional textbook based learning. Another advantage is the ability toreverse steps, which would obviously not be possible in the case of acadaver.

In another example, during examination of a horse, an augmented realitydisplay of equine anatomy can be displayed in the veterinarian's HUD,giving quick access to anatomical data and improving efficacy ofexamination and treatment.

Another embodiment relates to a method for combining gross anatomy withproblem based learning (PBL).

Gross anatomy is the study of anatomy through the use of cadavers orother anatomical teaching methodologies, while PBL is a pedagogy inwhich students learn about a subject through open-ended problems. Thetwo methods can be combined in order to create a learning paradigm inwhich open-ended problems are combined with anatomical dissection inorder to teach a more thorough understanding.

For example, an instructor could pose a problem involving a patient whohas passed away. In the hours prior to death, the patient repeated thesame question over and over, despite receiving an answer to the questioneach time. Students can then use a virtual body for dissection todetermine the cause of death, in this case an insulin-secreting tumourof the pancreas.

Another embodiment relates to a method for providing an augmented orvirtual reality medical simulation, comprised of elements including, butnot limited to, diagnostic simulations, surgical simulations, proceduralsimulations, previewing surgeries based on patient imaging, and groupsimulations for purposes such as teaching.

Medical simulations are useful for training and testing practitionerswithout risk to patients. Using data acquired from a real patient, orconstructed using a three-dimensional modelling program or through othercomputer-generated means, a patient is created in an immersiveenvironment.

A virtual patient can have a condition as selected either automaticallyby the software, or with user interaction for example by an instructor.The user or users can interact with the virtual patient in order todiagnose the condition. Virtual diagnostic tests can be run on thepatient, giving results accurate to the condition the patient isdisplaying.

FIG. 16 shows a virtual patient with a visible rash. The patient is tobe examined and diagnosed by the user.

A user can also perform a surgery or procedure, either as part of asimulation involving a diagnosis or separately. The virtual patientresponds as would a real patient, and complications can optionally beintroduced either automatically or interactively.

Surgical previews can also be performed using imaging data from realpatients. These images are transformed into a model usable by thesimulation, and a surgical procedure is simulated using the anatomy ofan actual patient.

For example, a simulation could begin with a virtual patient in adoctor's office. The user must question the virtual patient anddetermine the appropriate diagnostic tests for a diagnosis. In thisexample, the patient has pain in the lower back due to kidney stones. Inorder to diagnose this, the physician orders an abdominal MRI. In thesimulation, the results of the test are made available immediately.Using the MRI, the user correctly diagnoses the kidney stones and isable to schedule the patient for surgery. The simulation then moves to asurgical environment, and the user is able to perform the simulatedsurgery to treat the patient.

In another example, a surgeon preparing to install a pacemaker in apatient reviews the patient's radiological data in virtual reality. Amodel of the patient is constructed and placed on a virtual surgicaltable. The surgeon is able to use virtual surgical tools to install thepacemaker in the patient, using the real patient radiological data, inadvance of performing the actual surgery. This allows the surgeon toprepare for any abnormalities in the patient physiology as well aspractice the procedure for efficacy.

Another embodiment is a means of creating three-dimensionally printedcadaver models for anatomical use, surgical practice and other means.Using three-dimensional models created using the explained method fromradiological data, a model suitable for three-dimensional printing isgenerated. This model is of sufficient detail and accuracy to be used inplace of a cadaver for purposes of anatomical study, surgical practiceand other common uses. This also allows for printing of defective organsprior to surgical repair, which can be used for practice and study oftechniques. This also allows for problem based learning combined withgross anatomy in both real and virtual settings.

Three-dimensional models of animals and other organisms can also becreated, allowing for veterinary and other disciplines to performdissection and anatomical study on species that are either uncommon orotherwise difficult to study. An additional benefit of this method isthat the subject does not actually need to be killed. This isparticularly useful with endangered species, where a dissection is notpossible, but collection of radiographical imaging may be possible.

For example, radiological data from patients with tumours are used tocreate three-dimensional cadaver models for a classroom. Each cadaver isthen associated with a set of symptoms and radiological reports.Students must then correctly identify the issue, and perform thesurgical procedure on the cadaver to remove the tumour.

In another example, a man dies of unknown causes. The family does notwish an autopsy performed, however the police have questions regardingthe mans death. By scanning the body in an MRI, a three-dimensionalcadaver model can be created, which can then be autopsied withoutviolating the family's wishes.

Another embodiment relates to a method of using augmented or virtualreality combined with artificial intelligence for the purpose of testingand teaching materials to students.

Students learn in many different ways. Using artificial intelligence andan immersive environment, pre-programmed material can be presented to astudent in an engaging fashion. By continuously testing the students'knowledge of the subject material, the methods that are most effectivefor the particular student can be determined, and teaching can beaccelerated.

FIG. 17 shows a multiple choice question displayed in augmented realityfor a student.

The immersive environment can also be used for testing of thepre-programmed material. A student is asked to respond to questions, orto perform tasks, or otherwise interact with the immersive environmentas defined in the program. Based on the success or failure of theresponses, a grade can be assigned and areas of improvement can beidentified.

For example, a child with a learning disorder is introduced to avirtual- or augmented-reality learning environment. Information aboutdogs, cats and fish are presented in different fashions. Dogs are taughtusing visual cues. Cats are taught using audio methods. Fish are taughtusing an interactive display that can be touched and manipulated. Thechild is then tested to determine which portions of the material werelearned best. This is repeated over multiple topics, both to improveaccuracy and to account for cases in which the child has foreknowledgeof the subject area, and a learning profile is created and adapted forthe specific child. New material is then presented using the adaptedmethodology, and testing is used to continuously update the learningmodel.

In another example, dyslexia can be diagnosed using a series of wordsdesigned to test pronunciation and reading. Each word is presented inaugmented or virtual reality, and the user is asked to read the word outloud. Speech recognition is used to determine whether the word has beenrepeated correctly. Based on the number of words repeated correctly, anassessment can be made as to whether additional screening for dyslexiais required. The test can also be monitored remotely by another userwith a different device. This allows for testing without the subjectbeing anxious about being monitored during the test, helping them toperform better.

In another example, a student is given a test consisting of twenty-onequestions. Seven questions are given to the student verbally. Sevenquestions are given to the student visually. Seven of the questionsrequire the student to interact with virtual objects. The results of thetest are analyzed both for an overall grade, and for grades in eachindividual learning type. A profile for the student is built,determining if the student scores higher on questions posed in aparticular style. When a preferred style is determined, material will bepresented more often in the preferred format to assist the student inlearning.

In another example, a child with a learning disorder is introduced to avirtual or augmented reality learning environment. Information aboutdogs, cats and fish are presented in different fashions. Dogs are taughtusing visual cues. Cats are taught using audio methods. Fish are taughtusing an interactive display that can be touched and manipulated. Thechild is then tested to determine which portions of the material werelearned best. This is repeated over multiple topics, both to improveaccuracy and to account for cases in which the child has foreknowledgeof the subject area, and a learning profile is created and adapted forthe specific child. New material is then presented using the adaptedmethodology, and testing is used to continuously update the learningmodel.

Another embodiment is a means of performing a hearing test using anaugmented or virtual reality device. The test is performed by firstinstructing the user to indicate when they hear a sound. Sounds are thenplayed in increments, starting at a frequency well below normal humanhearing range, until the user indicates they can hear the sound. Oncethe sound is heard, the increment is reduced and the frequency isreduced until a sound is played and the user does not indicate hearingit. This is repeated until the lowest frequency heard by the user isfound. The user is then tested in the high frequency range, beginning ata frequency well above normal human hearing range. The frequency isdecremented until the user indicates that they can hear the sound. Theincrement is then reduced, and the frequency is increased until the userno longer hears the sound. This is repeated until the highest frequencyheard by the user is found.

For example, a child who is thought to be deaf is exposed to animmersive environment and connected to vitals monitoring. The child isthen exposed to various sounds, and the vital signs monitored. Aresponse by the child to the sounds indicates that they are able to hearthe sounds, and can be used to assist in diagnosis of conditions such asnon-verbal autism.

In another example, an aging woman is thought to perhaps be hard ofhearing. By having her perform the test, her auditory range can beverified and it can be determined whether she has a need for a hearingdevice.

Another embodiment relates to a method for augmented or virtual realitysimulation for the purpose of training a user or users in first aid.

First aid training can be done in an immersive environment usingpre-programmed simulations. A user interacts with three-dimensionalmodels in an immersive environment, following instructions given eitherby the computer running the simulation, or by a live instructor. Theinstructor, and other users, can optionally view the immersiveenvironment at the same time as the training user. Feedback is providedby the simulation. The simulation can also be used for testing andgrading of users.

FIG. 18 shows an augmented reality demonstration of a patient receivinga tourniquet. The demonstration is given by a virtual instructor,following which the user is invited to repeat the procedure.

For example, a group of students is learning to apply a tourniquet in afirst aid situation. A virtual reality program, complete with virtualinstructor, gives the group a demonstration of how the tourniquet istied. After the demonstration has been completed, each student is ableto attempt a tourniquet on their own virtual patient. When a student ishaving trouble, they can request assistance from the program. Whenstudents complete their tourniquet, the program evaluates their level ofcompetency and assigns a grade towards their first aid course.

Another embodiment relates to a method for doing intelligence quotienttesting using augmented or virtual reality.

IQ testing is frequently done in the presence of an examiner, which canmake some subjects nervous and affect performance. By administering thetest in an immersive environment, the user can take the test free of thedistraction of being watched. The administrator of the test couldoptionally watch the process in an immersive environment without beingvisible to the user.

The test is administered using the same test questions that would beused in a written/physical test, however all material is asked andanswered in an immersive environment. This also allows for more advancedtesting in areas such as spatial reasoning.

FIG. 19 shows a question posed for an IQ test in augmented reality.

For example, a test for spatial reasoning may involve a question ofwhich of a series of shapes will correctly fill a three-dimensionalhole. In augmented reality, the user is able to examine the shapes inthree dimensions, manipulating their orientation and size. This allowsthe user to better analyze the possible solutions to the problem beforemaking their selection.

Another embodiment relates to a method for teaching students usingaugmented or virtual reality combined with artificial intelligence.

By combining augmented or virtual reality and artificial intelligence,an enhanced learning system can be created for teaching of subjectmatter. Different people learn in different ways, with aural, tactileand visual being the three primary methods. By using artificialintelligence and a databank of information to be taught, the optimallearning style of a student can be gauged and utilized to ensure betterunderstanding of the teaching material.

By periodically assessing the student, the areas in which the studenthas not fully learned the material can be determined, and additionalteaching and focus can be provided on those areas. Using a combinationof teaching using different balances of the aforementioned methods, thestudents' best learning styles can be established either in whole or indifferent areas, and by adapting the teaching methods to the student,learning and retention are enhanced.

For example, a student who learns very well from written instruction isbeing taught how to perform a science experiment. Different parts of theexperimental method are imparted to the student using different teachingmethods: aural, tactile and visual. The program notes that the studentis best able to follow the instructions when they are presentedvisually, and therefore begins to present a higher proportion of theinstructions in a visual manner.

In some embodiments, a method comprises collecting first data from atleast one sensor used during a procedure, storing the data in memory,and replaying the data at a later time. Optionally, the methodadditionally comprises obtaining second data from the at least onesensor and comparing the second data to the first data. Optionally, themethod additionally comprises presenting an indication of the result ofthe comparison (e.g., to indicate whether the second data matches or isconsistent with the first data).

XVI. AR/VR-Assisted Neurological/Psychological Analysis and Treatment

Another embodiment relates to a method and apparatus for psychologicaldesensitization of phobias. Using an immersive environment, coupled withsensors to monitor vital signs, a patient's level of stress and fear ismonitored through a simulation. By exposing the patient to graduallyincreasing, yet tolerable, levels of phobic materials, the patient'stolerance is gradually increased. For example, a patient with a fear ofspiders is exposed to spiders in an immersive environment. This couldstart with simply showing a spider crossing the floor, and progress asfar as spiders climbing on the patient if the patient's vital signsindicate that the patient is not too stressed or fearful of theexperience.

As another example, a patient with agoraphobia may be exposed in animmersive environment to the experience of going out in public, startingwith leaving their house. Interactions with other people and otherstimuli are increased or decreased, depending on the monitored vitalsigns.

Another embodiment relates to a method and apparatus for psychologicaltreatment using a virtual person. In therapy, the concept of writing aletter to a person and not sending it is often used. By creating avirtual version of another person in an immersive environment such asaugmented or virtual reality, the user is able to talk or expressfeelings towards a person without them being present. This allows theuser to work out his or her feelings in a safe and comfortableenvironment, as outlined in an example below.

Another embodiment includes a method and apparatus for assistingpsychiatric and psychological patients using a reactive simulation inaugmented reality, virtual reality or other immersive environment. Forexample, a child who is thought to be deaf is exposed to an immersiveenvironment and connected to vitals monitoring. The child is thenexposed to various sounds, and the child's vital signs are monitored. Aresponse by the child to the sounds indicates that the child is able tohear the sounds, and can be used to assist in diagnosis of conditionssuch as non-verbal autism.

Another embodiment includes a method and apparatus for diagnosingpsychoses and phobias in patients using vital signs tracking combinedwith augmented reality, virtual reality, or another immersiveenvironment. Using stimuli in immersive environments while tracking thevital signs of a patient, phobias and/or psychoses can be evaluated. Inthe case of phobias, introducing a patient to situations involvingcommon phobias, such as heights or spiders, will cause elevated stressmeasurable in the patient's vital signs. In the case of psychoses, apractitioner can identify if a patient is reactive to virtual oraugmented stimuli, and to determine whether the patient believes thestimuli are real.

Another embodiment includes a method and apparatus for diagnosing traumavictims using vital signs measurements combined with augmented reality,virtual reality or another immersive environment. In therapy, a commonneed is to diagnose trauma suffered by a patient. Using virtual stimuliand vital signs measurements, situations that lead to elevated stressand fear in a patient can be discovered and analyzed by a practitioner.These situations will assist the practitioner in determining the sourceof trauma in patients. This is especially helpful with younger patientswho may be afraid to communicate their traumatic experiences, and withpatients who have no recollection of the events causing their traumas.For example, to determine whether a child has been abused, simulatedimages that may mimic situations similar to those experienced by thechild and monitoring the child's vital signs may be used. When thechild's vital signs indicate that the child is uncomfortable, stressed,or afraid, the practitioner can use the information to help guidetherapy for the traumatic experiences.

Another embodiment includes a method and apparatus for diagnosingepilepsy using EEG or MEG and vitals sensing, light events, and/or otherstimuli in an augmented reality, virtual reality, or other immersiveenvironment.

Another embodiment includes a method and apparatus for determiningresponses to virtual stimuli, detecting fabrications in stories, andother vital signs detection. When a person lies, their pupils dilate orthey look in specific directions. Using a camera or other monitoringdevice to watch a subject's eyes, a lie can be detected. Responses tostimuli can also be determined from vital signs monitoring. Elevatedheart rate, blood pressure, pupil dilation, eye movement, eye direction,and/or other vital signs can be used to determine how a subject feelsabout a given stimulus. Virtual stimuli can be shown to a subject in animmersive environment, and the resulting vital signs analyzed todetermine the subject's feelings regarding the stimuli.

Another embodiment relates to a method for using augmented or virtualreality for psychological desensitization of phobias. As one example,the vital signs of a user who has a phobia (e.g., a fear of spiders) canbe monitored to determine the user's level of stress and either increaseor decrease exposure to a trigger (e.g., spiders) in an immersiveenvironment to help the user overcome the phobia. FIG. 20 shows avirtual spider coming out of a hole in the wall, as well as the vitalsof the user in the HUD.

In some embodiments, an immersive environment is used to simulateaspects of a user's phobias. Virtual models or situations simulating thephobia to be used are pre-generated for use in the immersiveenvironment. The user is connected both to a viewing medium for theimmersive environment, and to monitors for vital signs. The user'spulse, blood pressure, and/or other key vital signs are monitored forchanges. When the user is exposed to a phobia, the change in vitals ismeasured. If the change is larger than a given threshold, then theexposure to the phobia source is reduced. If the vitals remain within anacceptable range, the exposure is gradually increased. Over time withexposure, this will assist the user in dealing with the phobia inquestion.

For example, a user who is scared of heights wears a set of virtualreality glasses. A safe environment is shown to the user to gatherbaseline vital signs as well as to allow the user to adjust to theimmersive environment. Once the baseline reading has been gathered andthe user has adjusted to the immersive environment, the user indicateshe or she is ready to begin. The user is shown a view from the top of ahill. The vital signs show that the user is slightly uncomfortable, butwithin an acceptable range of discomfort. The user is given time to lookaround the immersive environment at this height, and to adjust to theheight being shown. As the vital signs return towards the baseline, theenvironment is shifted to one at a greater height. This time, the useris too uncomfortable with the height, as indicated by the user's vitalsigns. The environment is shifted back to the previous height, allowingthe user to calm down. If this height is insufficient to calm the user,a lower height or safe environment can be used and the process startedover again using a more gradual increase in heights. As the usercontinues the program, his or her tolerance to greater heightsincreases, helping the user to deal with the phobia.

Another embodiment relates to a method and apparatus for psychologicaltreatment using a virtual person. The virtual person is an avatar. Insome embodiments, the virtual person is programmed to respond to theuser. For example, a user who has trouble with a supervisor at work cancreate a virtual supervisor to whom the user can talk about the user'sproblems. In another example, a user can speak with the avatar of aloved one who has passed away, allowing the user to find closure.

Another embodiment relates to a method for assisting psychiatric andpsychological patients using a reactive augmented or virtual reality.For example, a patient with alcoholism has successfully completed arehabilitation program. After leaving a treatment facility, the patientneeds psychological help in order to reintegrate with normal society.Using a virtual reality environment, the patient is exposed to realworld situations, the patient's reactions are gauged, and assistance andguidance given, either automatically or with assistance from anotheruser. The patient learns through experience how to relate and behavewith normal societal circumstances, but is able to do so while remainingin a safe and controlled environment.

In another example, a patient suspected of having schizophrenia can betested by presenting the patient with an augmented reality environmentshared with a therapist. Frequently, schizophrenic patients will lieabout hearing and/or seeing things that are not there, which can makediagnosis difficult. The therapist can initiate sounds and sights in theaugmented reality environment, and ask the patient questions about whatthe patient sees and/or hears. The patient's responses are analyzed bythe therapist, helping the therapist to determine whether the patient isschizophrenic.

Another embodiment includes a method for determining psychosis andphobias in patients using vital signs tracking combined with augmentedor virtual reality stimuli. Phobias can be identified in simulation froma database of common phobias. In an immersive environment, a user isattached to vital signs monitoring. The vital signs are in turn fed intothe simulating computer, which controls the immersive environment toexpose the user to a variety of potential phobias. By monitoring therespondent vital signs, a user's level of comfort with each potentialscenario is determined. This data can be compiled into a phobia profilefor use by a therapist or other professional.

For example, a user connected to a vital signs monitoring system beginstesting by wearing a set of virtual reality glasses. The simulationbegins with a safe place, such as a simple nondescript room. Baselinereadings are taken for the user's vital signs. To test whether the userhas various fears or phobias, different stimuli are added to theimmersive environment, and changes to the user's vital signs aredetected and/or recorded. For example, to test whether the user isafraid of mice, while the user looks around the room, a mouse emergesfrom a hole in the wall of the immersive environment. The user's vitalsigns are monitored for a reaction to the mouse and recorded. If theuser's vital signs remain unchanged, that indicates the user likely hasno fear of mice. In this case, the mouse may return to the hole anddisappear. To test whether the user has a fear of spiders, a spideremerges from the hole, and again the user's vital signs are monitoredfor a reaction. Assume for the sake of example that the user's vitalsigns change in a way that indicates mild distress. The vital signs arerecorded, and the spider returns to the hole. To test whether the userhas a fear of bats, a bat is introduced to the scene, flying in througha window. Assume for the sake of example that in response to the bat,the user's vital signs rise sharply, indicating acute distress. FIG. 21shows a simulated bat, and elevated vitals related to a user's fear ofbats. The readings are recorded, and the bat is immediately removed fromthe scene. The readings resulting from the introduction of stimulirepresenting any number of phobias can be recorded for later inspection,printing, display, or electronic transmission.

Another embodiment includes a method for diagnosing trauma victims usingaugmented or virtual reality combined with vital signs measurements inorder to determine sources of potential past or current traumas. In someembodiments, a user in an immersive environment simulation of a varietyof events can be connected to vital signs monitoring, which is in turnconnected to the computer system running the simulation. By monitoringthe vital signs of the user, it can be determined what situations makethe user anxious or uncomfortable. The simulation scenarios arepre-programmed and can form a database of possible scenarios. Thescenarios can either be selected automatically, or with user input bysomeone like a psychologist or other professional. The measured vitalsigns and interpreted reactions can be recorded to a storage medium forfurther consultation.

As just one example, FIG. 22 shows the view of a user whose vital signshave been elevated due to the presence of a stranger. As anotherexample, children are inclined to lie about abuse, be it physical,sexual or emotional when they have been threatened or have been abusedby someone they care about. A child who may have been abused by his orher father may be connected to vital signs monitoring and don a set ofvirtual reality glasses. A simulation begins with a safe, nondescriptroom to allow a baseline reading of vital signs to be gathered andrecorded. The child is then shown virtual images of people with whom thechild lives. First, the child's sister is shown. The simulated sistermay be shown exhibiting a variety of emotions, such as sadness,happiness, anger, and fear. The child's reaction to each is noted andrecorded. This process is repeated for each member in the child'shousehold. When the child is shown an angry simulation of the father,the vitals may indicate extreme fear and distress if the child has infact been abused by the father. The results of the test are recorded andstored for review by a qualified professional.

Another embodiment includes a method for diagnosing epilepsy using EEGor MEG and vitals sensing, light events and other such stimuli invirtual or augmented reality to monitor responses.

Another embodiment includes a method for using pupil dilation, eyemovement and eye direction for determination of response to stimuli,detecting fabrication in stories, and/or other such vital signsdetection. A camera or other sensor targeting the face of a subject isused to detect dilatory changes in the pupil size, position, andmovement. FIG. 23 shows a visible difference in pupil dilation. Thesensor data is analyzed to determine whether dilations are a result ofchanges in the environment, or a result of changes in the subject'sstate of mind. For example, ambient light levels, particularly thosedirectly around the eyes of the subject, are used to determineenvironmental lighting factors.

As an example, a suspect is being interrogated by a police officerfollowing an armed robbery. The police officer is wearing a set ofaugmented reality glasses that are programmed to analyze the suspect'sface as described above. As the suspect tells a story of having beennowhere near the location of the robbery, the software indicates thesuspect's pupils dilate abnormally, as well as eye movement, bodyposture, and/or sweating indicative of dishonesty. This gives the policeofficer a strong indicator that the suspect is lying, and leads to moredirect questioning regarding the circumstances of the crime.

In another example, a practitioner using a set of augmented realityglasses can examine a subject for neurological conditions, such asstroke or other neurological symptoms. Additionally, audio can berecorded and/or interpreted to determine if the subject has a slur orother auditory symptom.

In some embodiments, a method comprises creating an environment inaugmented or virtual reality for a user, monitoring at least one vitalsign (e.g., a heart rate, a blood pressure, a pupil dilation, an eyemovement, an eye direction, etc.) of the user while the user is immersedin the environment, and adjusting an aspect (e.g., changing an aspectrelated to phobia of the user, changing a perceived distance between theuser and a virtual object within the environment, adding a virtualobject to the environment, removing a virtual object from theenvironment, etc.) of the environment based at least in part on themonitored at least one vital sign. For example, the adjustment may bemade in response to a change in the monitored vital sign. When theadjustment is to add a virtual object to the environment or remove avirtual object from the environment, the virtual object may be a virtualversion of a person that may be capable of interacting with the user.The method may further comprise providing guidance to the user while theuser is immersed in the environment. The method may further compriserecording the at least one vital sign. The method may further compriserecording a response of the user to a change in the environment whilethe user is immersed in the environment. The method may further comprisegenerating a profile based at least in part on the monitored at leastone vital sign of the user while the user is immersed in theenvironment. The method may further comprise connecting the user to amonitoring device (e.g., a heart rate monitor, a blood pressure monitor,an imaging device, etc.), and monitoring the at least one vital sign byobtaining information from the monitoring device.

In some embodiments, the method further comprises storing, in memory, anindication of an effect of the adjusting the aspect of the environmenton the monitored at least one vital sign. The method may furthercomprise identifying a phobia or psychosis based on the indication,and/or comparing the indication of the effect to information stored in adatabase (e.g., information identifying a phobia or a psychosis).

It is to be understood that in addition to diagnostic activities, theembodiments disclosed herein are also useful for training, education,and consulting activities.

XVII. AR/VR-Assisted Visualization

Another embodiment relates to a method and apparatus for providing avisual display for cosmetic surgical use in augmented reality, virtualreality or other immersive environment. The display can also be used tocreate models suitable for printing in two and three dimensions.

Cosmetic surgeons use various techniques to show patients and potentialpatients the possible results of cosmetic procedures. Using an immersiveenvironment, a cosmetic surgeon can show a subject how the results ofthe procedure will look in real-time, on the actual patient rather thana mock up or image. The virtual model can also be used to verify that asurgery is done correctly. The model can also be used during surgery tohelp the surgeon ensure a correct result. The model can also be used toshow the subject that the results are as expected.

In AR, the cosmetic surgery overlay is comprised of an augmented realitydevice or similar display, a camera or other imaging device, andoptionally an audio capture device or other audio input device. In VR,the overlay is comprised of a VR device and optionally an audio capturedevice or other audio input device.

In AR, the surgeon begins by selecting a region of the patient to becosmetically altered. The selection is made by gesture, voice command orother user input method. The selection can be fine-tuned using similarinput methods in order to ensure that the correct area is selected. Thesurgeon can then finalize the selection, and begin creating a new modelfor replacement. The surgeon uses the existing selection as a basis forthe new model, or optionally uses a pre-generated three-dimensionalmodel of the appropriate body part from which to model the desiredresult. This model is also optionally created from morphology data asdescribed below.

In VR, a model of the patient is used as a basis for selecting andmodifying the model. The model is adjusted and shaped in situ, insteadof as an overlay over the physical patient.

Modeling is done through a series of user inputs, allowing the surgeonto adjust the appearance of the three-dimensional model in order toarrive at the desired shape. The three-dimensional model is comprised ofpoints, which can be selected and manipulated individually or in groups.The selected area, is then adjusted in size and shape until it has anappearance satisfactory to the patient and surgeon. The augmented orvirtual reality environment is also optionally shared between thesurgeon and patient, each equipped with their own AR or VR device.

The use of a three-dimensional model generated in this way is usedduring surgery using the same methodology outlined in section I forsurgical overlays. This allows the surgeon to verify the correctness ofthe outcome against the expected model agreed upon with the patient,thereby reducing surgical error, helping avoid malpractice issues, andimproving patient satisfaction.

Post surgery, the surgeon can use augmented reality to show the patientthat the surgical outcome matches the expected model by once againshowing the patient the model overlaid over their features. Thisdemonstrates to the patient that the surgery has been completedcorrectly.

An example of surgical assistance is in breast augmentation. Many womenhave breasts of two different sizes, which requires the surgeon to guessat the correct adjustment to the implant sizes. Using a virtual overlay,the surgeon can ensure that the correct adjustment is made in real-time,and visually verify the results.

Another embodiment relates to a method and apparatus for displayingorthodontic images in virtual reality, augmented reality or otherimmersive environment.

Current practice in orthodontics involves taking x-rays and moulds of apatient's teeth, as well as jaw and bite measurements, and sending thedata to a lab to interpret the data and create dental appliances.

A virtual overlay can be used in orthodontics to show patients andpotential patients the results of orthodontic work. A virtual overlaycan also be used to determine the shape and sizing of a dental device.The overlay can also be exported in a format suitable forthree-dimensional printing of a dental appliance.

AR creation of dental models is comprised of an augmented reality deviceequipped with a camera or other imaging device, and optionally an audiocapture device for voice input and/or recording.

Creating the dental model is done by first initializing the system,using a voice activation or other form of user input. Once initialized,the system creates a point cloud of the observed data, in this case thepatient's teeth. The user or users look at the teeth from as many anglesas possible to ensure completeness of the point cloud. The user(s) alsohave the patient close their mouth in order to view track the alignmentof the teeth as they bite together. The recorded point cloud is thenused to create a three-dimensional model of the teeth and jaw, which canbe either sent to a laboratory for manufacture of dental devices, orused by the user or users themselves to generate a dental appliance forthree-dimensional printing.

In the case of a three-dimensionally printed dental appliance, astandard appliance model is fitted over the model of the teeth. Themodel is then adjusted by the user or users, using voice, gesture orother means of user input, to create a model that provides theappropriate correction to the alignment of the teeth. Once the desiredshape has been created, the appliance model is saved forthree-dimensional printing or sent directly to a three-dimensionalprinter for manufacture.

Another embodiment relates to a method for providing an augmented orvirtual reality view for cosmetic surgical usage, comprised of elementsincluding, but not limited to, patient previews, verification ofresults, and assistance during surgery.

In cosmetic surgery, expected results are typically generated usingsoftware designed specifically for that use. Using an immersiveenvironment, projected patient results can be manipulated and displayedin real time allowing a practitioner to show a subject how a particularsurgical alteration will appear when completed directly on their ownbody. This could be done using augmented reality and a mirror, usingvirtual reality and a model of the patient taken by any known method foracquiring said image, or other immersive environment allowingsuperimposition or alteration of the appearance of the subject.

FIG. 24 shows a doctor and patient looking at a possible new nose,overlaid in augmented reality. The new nose overlay is semi-transparent,allowing the old nose to be seen through it.

Using gestures, voice or other means of control, a practitioner canmodify the surgical results in an immersive environment. For example,with a rhinoplasty, the surgeon could control the simulation to show thesubject how they would look with different noses.

FIG. 25 is a flow chart walking through the process of creating andadjusting a surgical overlay. The process begins at the cell labelledstart. The first step is for the user to select the model source,whether it be a pre-created model from a databank or other data source,or a selection from the subject's existing anatomy. If a pre-createdmodel is selected, the process continues from location selection. If asubject anatomy source is selected, the user selects the area to be thebasis for the creation, selecting the area using a gesture or other userinput command. If the target area is incorrect, the user can adjust theparameters (such as the width, height, depth and location) of theselection in order to select exactly the area desired. The user thenconfirms the target area, and the process continues from locationselection. Next, the user performs location selection by placing the newmodel in the correct location on the subject's body. Once the model isplaced, the user then determines if it appears correct. If the modelappears incorrect, the user can select all or a portion of the model foradjustment. The user then adjusts the selected portion's size,orientation and location to the suit. The user can cancel the option ifthe new appearance is not satisfactory, in which case the model is resetto it's previous state and the portion selection is started again. Theuser can also confirm the changes to the model, at which point the useris able to decide once again whether the model is correct. Thiscontinues until the user is satisfied that the model is correct, atwhich point the model is saved for later use, and the sequence ends.

The model data can also be used in an immersive environment during theactual procedure, allowing a practitioner to be guided by the results.This allows for more accurate results and reduced surgical errors. Themodel data can also be used following the procedure in order todemonstrate to a subject that the results are as expected.

FIG. 26 shows a rhinoplasty model overlay during a cosmetic surgicalprocedure. The replacement nose shape and dimensions are shown intransparent grey. The model is transparent so as not to interfere withthe surgeon's view of the procedure.

For example, a potential patient doing a consultation for rhinoplasticsurgery could discuss options with the surgeon. The surgeon could thencreate a sample nose for the patient in augmented reality, and thepatient could review the potential results by looking at themselves in amirror using augmented reality.

Another embodiment relates to a method for providing an augmented orvirtual reality display for orthodontic use, comprised of the ability todisplay previews of orthodontic work, a method for showing future toothalignments and positions, a method of determining shapes and sizes ofdental devices, and a method of generating data files of dental devicesfor 3D printing.

An orthodontic patient or potential patient can be measured using athree-dimensional (depth) camera or other imaging device. The patientcould also be measured using traditional means, or other means ofmeasurement. Using these measurements, a three-dimensional model of thepatient can be created for use in an immersive environment. With thethree-dimensional patient model, dental appliances can be created in animmersive environment. These dental appliances can then be exported fromthe simulation in a format suitable for three-dimensional printing orsubmission to a manufacturer of dental appliances. For example, apatient looking to have orthodontic work can have a virtual realitysimulation of their teeth shown to them, displaying the changes in theirteeth alignment over time in a simulated environment.

In some embodiments, a system comprises a rendering device, and animaging device coupled to the rendering device and configured to provideimages to the rendering device.

In some embodiments, a method comprises creating an immersiveenvironment, and, within the immersive environment, creating a virtualmodel representing a result of a procedure (e.g., a cosmetic surgeryprocedure, a dental procedure, an orthodontic procedure, etc.) to beperformed on a patient, wherein the virtual model is a three-dimensionalmodel, and storing information representing the virtual model in memory.Creating the virtual model may comprise selecting a region of thepatient to be modified and creating the virtual model using the selectedregion of the patient as a starting point. Alternatively or in addition,creating the virtual model may comprise selecting, from memory, arepresentation of a body part and creating the virtual model using therepresentation of the body part. Alternatively or in addition, creatingthe virtual model may comprise obtaining a practitioner input (e.g., avoice command, a gesture, a selection through a virtual peripheral ordevice, a keystroke, etc.) and creating the virtual model based at leastin part on the practitioner input. In some embodiments, the virtualmodel comprises a plurality of points, and creating the virtual modelcomprises setting or modifying at least one of the plurality of points.The method may further comprise referring to the virtual model duringthe procedure performed on the patient. The method may further comprisemanufacturing an apparatus (e.g., a dental appliance) based on theinformation representing the virtual model, wherein the apparatusinstantiates the virtual model.

In some embodiments, a method comprises obtaining a first user inputidentifying a model source (e.g., a library of candidate models, thepatient's body, etc.), the model source providing a model (e.g., aselected model from a library of candidate models, a three-dimensionalrendering of a portion of the patient's body, etc.) for use in aprocedure performed on a patient, obtaining a second user inputidentifying a target area, obtaining a third user input indicating aposition of the model relative to the target area, obtaining a fourthuser input representing an instruction to modify at least an aspect ofthe model (e.g., a size, dimension, volume, area, orientation, location,or placement of the model), creating a modified model based on theinstruction to modify the at least an aspect of the model, and storingthe modified model in memory. In some embodiments, the method furthercomprises obtaining a fifth user input canceling the instruction tomodify the at least an aspect of the model. The method may furthercomprise canceling a modification to the model in response to the fifthuser input. In some embodiments, the method further comprises obtaininga fifth user input confirming an accuracy of the model or the modifiedmodel. In some embodiments, the method further comprises obtaining afifth user input comprising an instruction to save the modified model.

XVIII. Physiological/Anatomical Mapping, Modeling and Positional Marking

Another embodiment relates to a method and apparatus for scanning,mapping and analyzing human bodies.

Using a camera or other visual recording device, a subject can bescanned and mapped into a two- or three-dimensional model. This modelcan be used by a practitioner to identify areas of interest or concern.The model can also be used to monitor areas between visits. The modelcan also be analyzed, automatically or with user-interaction todetermine the presence of conditions such as melanoma, rashes, psoriasisand other visible conditions.

In the case of a two-dimensional mapping, a camera is directed at thesubject. The subject then turns 360 degrees, and images are recorded asthe subject turns. The recorded images are first processed to remove thebackground by comparing identical data from one frame to the next.Identical data is discarded, leaving only the subject. Using featuredetection, the images are then stitched together to form atwo-dimensional model of the subject.

In the case of a three-dimensional mapping, a camera is directed at thesubject. The subject then turns 360 degrees, and images are recorded asthe subject turns. The recorded images are first processed to remove thebackground by comparing identical data from one frame to the next.Identical data is discarded, leaving only the subject. A two-dimensionalmodel is created as explained above. A point cloud is then generatedfrom the data, creating a three-dimensional model of the subject. Thepoint cloud is then overlaid with the two-dimensional model (“skin”),which gives a three-dimensional model of the subject.

Once the model has been created, analysis of the two-dimensional model(“skin”) is performed for known conditions. Areas of interest are markedfor review by the user or users. The data is also stored for comparisonupon future visits.

Another embodiment includes a method for mapping and analyzing humanbodies, comprised of scanning of the body, storing of surface data,marking of important features such as melanoma, moles, rashes, otherskin conditions and remarkable features (either automatically or byhuman interaction).

A subject or subject area can be scanned using a camera or other imagingdevice. The surface data can then be stored and analyzed for current andfuture use. By analyzing the characteristics of the surface, commonconditions can be diagnosed, and efficacy of treatments can bedetermined. Sizes, colour and other metrics of an affected area can bemeasured and compared, allowing a direct comparison between previousvisits and current visits. This comparison also gives a clear view ofthe efficacy of treatments being provided. These comparisons can be usedby, but are not limited to, the practitioner as well as, for example, aninsurance company to determine whether they're willing to continuereimbursing the patient for a given treatment.

For example, a visual recording of a patient is taken with augmentedreality glasses is stored, complete with a visual overlay of diagnosesmade either automatically or with user-interaction. This recording canthen be used as a visual report for the patient file, and for reviewprior to appointments with the patient. The recording can also be usedas part of a referral to a specialist (including all AR/VR content). Therecording can also be used as part of a referral to a specialist. In ahospital setting, the visual record can be used to prevent the need tore-examine a patient at different stages of their treatment. A recordingof the original exam can therefore be viewed.

In another example, a patient with eczema could be scanned at an initialconsultation. As the dermatologist treats the eczema using aprescription, the scan can be compared at each visit to verify theefficacy of the treatment. Software can automatically determine whetherthe size and shape of the affected area has changed.

Another embodiment relates to a method and apparatus for timing thepulse sequences of MRI based on the position of the subject's body inorder to ensure that images are taken at the same point in a rhythmicmovement such as breathing or a beating heart.

MRI with a traditional MRI machine is subject to imaging problemsrelated to patient movement. Blurred images and image artifacts are twocommon issues seen when a patient moves during an MRI exam. Bymonitoring the position of the patient's body, the imaging sequence canbe timed such that an image is taken only when the patient is in thecorrect position.

For example, a sensor or camera can be used to monitor the height of apatient's chest, triggering the imaging sequence to take an image onlywhen the chest is at the same height as the last image. Using thistechnique, all images of a patient's chest would be taken when the chestis in the same position.

Another embodiment includes a method and apparatus for interpreting rawMRI signal data into composite three-dimensional models.

When an MR image is taken, it is recorded as a series of signals asrecorded by receivers in the MRI machine. These receivers measure themagnetic resonance of the subject at the time of recording. Usingmultiple receivers gives a large number of data points that need to bebroken down into parts in order to generate images. By interpreting thedata, a three-dimensional model of the subject can be created suitablefor virtual reality, augmented reality and three-dimensional printingapplications.

Another embodiment relates to a method for enhancing positional locationin augmented reality using gadolinium markers.

Another embodiment relates to a method and apparatus for controlling thevisualization of a three-dimensional object in virtual reality,augmented reality, or other immersive environment.

A three-dimensional object stored in a computer consists of many datapoints. By altering the visualization, the visual representation of theobject can be changed, allowing a user or users to view the visualizedobject in different ways.

For example, a three-dimensional model created from MRI data contains agreat deal of information that is covered by the outer layers of themodel. By altering the visualization and removing the outer layers ofdata, the inner portions of the model (such as the brain) can be madevisible.

Another embodiment relates to a method and apparatus for visualizingmedical imaging data in augmented reality, virtual reality, or otherimmersive environment.

Medical imaging data can be converted to a format suitable for displayin three-dimensional virtual space. This data can then be displayedthrough virtual reality, augmented reality, or another immersiveenvironment.

Positional location in augmented reality is determined primarily throughvisual means, feature detection, and other methods described herein.

Another embodiment relates to a method and apparatus for constructing athree-dimensional model comprising the steps of determining imageseparation distance, identifying missing images, aligning source imageand constructing missing image data, and merging the images to form athree-dimensional model.

Another embodiment relates to a method and apparatus for detecting andmonitoring a user's hands, or another part of the user's body, inaugmented reality, virtual reality, or another immersive environment.

For example, a set of one or more sensors attached to the user's handscan be used to determine the position of the user's hands relative tothe virtual camera. The positions of the one or more sensors can be usedto determine the positions of individual segments of the user's hands,such as, for example, the user's palm, fingers, and wrists. Data istransmitted from the sensors to the immersion device controlling theimmersive environment.

Another embodiment relates to a wearable apparatus for full-body sensingand feedback. In some embodiments, a system comprises one or more ofmeans for measuring and tracking the wearer's movement, means forsimulating touch senses, means for simulating temperature senses, or ameans for restricting user movement.

Another embodiment relates to a wearable apparatus for full-body sensingand feedback. In some embodiments, a system comprises one or more ofmeans for measuring and tracking the wearer's movement, means forsimulating touch senses, means for simulating temperature senses, ormeans for restricting user movement used for gaming in augmented orvirtual reality.

Another embodiment relates to a method and apparatus for recording andreplaying sensory data using sensors.

In some embodiments, a user wearing a glove or other device equippedwith at least one sensor touches a surface, and the at least one sensorrecords characteristics of the surface detected from the touch. Thecharacteristics may include, for example, the texture of or pressurepresented by the surface. A processor coupled to at least one actuatormay later use the recorded characteristics to control at least oneactuator such that the at least one actuator emulates the feeling of thesurface. By taking multiple recordings, either from the same surface ordifferent surfaces of the same type, a profile can be created for agiven surface. Using profiles generated in this way, surfaces can beidentified through touch by comparison to existing profiles.

For example, a surgeon wearing a glove having at least one sensor cantouch a human limb while the at least one sensor records, for example,the resistance presented by the human limb. The processor can then usethe recorded data from the at least one sensor to control at least oneactuator to emulate the feeling of touching the recorded limb.

Another embodiment relates to a method for timing imaging sequencesbased on position of the patient's body, for example using the height ofthe chest to ensure that images are taken at the same point during thebreathing process to give a more stable image.

In traditional imaging sequences, movement of the patient can causefailed imaging sequences, artefacts, blurred images, and/or otherundesirable anomalies. By using a sensor, for example a camera,altimeter, or other positional sensor, the imaging sequence can be timedto take images only when the patient is in the correct position.

For example, in doing an MR scan on a patient's chest, a camera can beused to monitor the height of the patient's chest from the MR platform.When the patient's chest is at a specific height, the imaging sequenceis fired. When the patient's chest is no longer at the correct height,the sequence is paused awaiting the next time that the chest position iscorrect.

FIG. 19 shows a patient in an MRI machine. An imaging line is shown inthe image, which is a line tracked by a camera or other imaging deviceat which images are taken. When the patient's chest is not level withthe line, imaging is not taken.

Another embodiment includes a method for interpreting raw MRI signaldata into composite three-dimensional models for use in virtual reality,augmented reality, and/or three-dimensional printing applications.

Raw MRI signal data can be composed in many different fashions. Byinterpreting the raw signal data from a MR scan, a three-dimensionalrepresentation of the target area can be created. The raw signal isfirst decoded into voxels using methods common in the industry. Thesevoxels are then translated into three-dimensional coordinate spacewithin the computer. Using this three-dimensional voxel model, athree-dimensional model can be created for applications such as animmersive environment simulation and three-dimensional printing.

Another embodiment includes a method for controlling the visualizationof a three-dimensional object displayed in virtual reality, augmentedreality, or other immersive environment comprising the steps ofdetermining the requisite change in visualization, and updating thethree-dimensional object. An apparatus for controlling the visualizationof a three-dimensional object displayed in virtual reality, augmentedreality, or other immersive environment comprising a means ofdetermining the requisite change in visualization, and a means forupdating the three-dimensional object. The process may be performedautomatically by a system or may be guided interactively by an operator.Applications include, but are not limited to, virtual reality, augmentedreality and three-dimensional printing.

A visualization in an immersive environment can be controlled in avariety of different ways in various embodiments. In one embodiment, themodel display depth is controlled automatically or by user interactionto display parts of the model not initially visible. The model caneither be densely packed (including all information) or a “hollow” modelconsisting of perimeter information only to a limited depth. Thisperimeter information can be calculated using negative spaceexploration. As the user indicates a portion of the model they wouldlike to see deeper, the outer sections of the model are hidden and theunderlying data is displayed.

Negative space exploration is don e by selecting an empty starting pointat the edge of the model's cartesian space, frequently at (0, 0, 0) [x,y, z coordinate]. Each adjacent coordinate is added to an explorationlist provided that the coordinate does not satisfy the search parameter,for example a minimum colour value threshold. When a point is met thatsatisfied the search parameter, it is added to the objects perimeterarray, and in the case of depths greater than one coordinate the depthcounter for the angle is decremented. Coordinates satisfying the searchparameter are not added to the search list.

FIG. 28 shows an example of two-dimensional negative space exploration.The exploration started from the point (0, 0) in the top left corner.Points were added to the searched area (see legend) and adjacent pointstested for non-zero (white) values. Along the top left perimeter of thecircle (Image, see legend) non-zero points have been found (Perimeter,see legend). These points satisfy the non-zero search parameter and areadded to the perimeter array. Therefore, as of the point in timedepicted in this figure, the perimeter array contains the points: (8,3), (8, 4), (7, 4), (6, 4), (6, 5), (6, 6), (5, 6), and (5, 7).

In the case of updating a hollow model, data from the complete model isused to determine the data to be displayed at the new depth location.For example, if the initial depth along the x-plane is 0, and the userhas updated the depth to be 10, all coordinates in the existing modelwith an x-value less than 10 are discarded from the model. Data from thecomplete model is then added along the x=10 plane of the model.Additionally, data to a given depth can be added. For example, if thedepth to be used for the model is 3, data in the range 10<=x<=13 wouldbe added to the visible model.

Another embodiment includes a method for visualizing medical imagingdata in augmented reality, virtual reality, or other immersiveenvironment, comprising the steps of locating the subject, determiningsubject position, determining subject orientation, and rendering themedical imaging data. An apparatus for visualizing medical imaging datain augmented reality, virtual reality, or other immersive environment,comprising a means for locating the subject, a means for determiningsubject position, a means for determining subject orientation, and ameans for rendering the medical imaging data. The process may beperformed automatically by a system or may be guided interactively by anoperator. Applications include, but are not limited to, visualizationfor the purpose of surgical procedures, visualization for the purpose ofmedical testing, visualization for the purpose of surgical training,visualization for the purpose of medical training, visualization for thepurpose of physiotherapy, visualization for the purpose of lasersurgery, and visualization for the purpose of physical diagnostics.

Locating a subject can be done in a variety of different ways. In oneembodiment, features in the subject area are compared to featuresdetected in the target. If the number of matching features is greaterthan a threshold, determined either automatically or through user orprogram specification, then the target is deemed a match to the subjectand the match location is found. In another embodiment, the perimetershape of the target can be compared to detected edges in the image. Ifthe number of matching perimeter points exceeds a threshold, eitherautomatically determined or specified by a user or program, then thetarget is deemed a match to the subject and the match location is found.This process can be applied in three dimensions using, for example, apre-compiled set of features or perimeter data for different angles andscales of the target. Additionally, the rotation and scale of the targetcan be determined in real-time during feature or perimeter comparison.

FIG. 29 shows a target object (bottom left corner, white background)being located and matched in an image. The white X marks on the targetobject indicate features. These features are matched to features in thesubject image to find a positive identification. The perimeter values ofthe target object are also compared to the subject image to find and/orreinforce the match. The matching area is shown with a black squaresurrounding it.

The search area within the subject can be further reduced in order tomake detection of targets a faster process. One embodiment uses an XOR(exclusive or) method to determine points in the image that havechanged, indicating movement of objects within the subject. These motionpoints are used to guide the search for targets in the subject, reducingthe number of data points that need to be examined. These points canoptionally be used as replacements for features and/or perimeter data.

In order to determine the XOR based image, the offset between frames isrequired. To determine the offset between frames, the current subjectimage is compared to the previously seen subject image. A number ofpoints (n) is selected either automatically, by a user, or as a part ofthe program. These points are fixed locations in the view frame. Bycomparing the data in the previous image to the data in the currentimage, an offset can be determined. One point is selected as a startingpoint. An area, either predetermined, automatically determined, orselected by a user, is searched for a match to the value of the previousimage. The value to be compared can be, for example, a single pointvalue. The value can also be the summation of a Gaussian distribution orother means of calculation. If the value in the current image is foundto match the value of the previous image within the given range, thenthe offset is recorded. Other possible offsets within the range are alsorecorded. If no possible offsets are found, then another point isselected until either a match is found, or a subsequent match for theoffset (see below) is no longer possible.

FIG. 30 shows a flow chart for frame offset calculation. The flow beginsat Start in the top left corner. If this is the first frame of thesequence (e.g., First image captured from a camera), we simply save thecurrent frame and complete the sequence. If this is any subsequentframe, we store the previous frame and add the current frame. Next, anumber of reference points (N) are selected, either at predefinedcoordinates or by some other means of selection. These referencecoordinates are used to retrieve values from the previous frame. Thevalues are stored for later use. The values at the reference coordinatesin the current frame are then compared to those taken from the previousframe. If a sufficiently high number of values do not match, then atransformation of coordinates will occur. First, the transformationvalues are tested to ensure they haven't exceeded thresholds. If theyhave, the sequence is aborted and no match is found. If they have not,then the translation and/or rotation values are adjusted in a logicalfashion to test values within the threshold ranges. The cycle ofcomparison and adjustment is continued until either the transformationthreshold is exceeded and the sequence ends without a match, or asufficiently high number of values do match and the rotation andtranslation values are recorded. Using the recorded translation androtation values, the previous frame and current frame are then combinedusing an XOR operation, giving a new frame of the same size as theoriginal frames. By finding coordinates within the XOR'd frame thatexceed a given threshold, the locations of objects and other movingcomponents of the image become visible.

Once the list of possible points is completed, each of the remaining npoints is compared at the same offset. These points are also rotatedbased on the center of the image and tested. If enough of the pointsmatch at the specified offset and rotation, a match is determined to befound. At this point, all of the pixel values in the target image areXOR'd with the subject image, modified by the determined offset androtation. Points that do not exceed a threshold (either determined by auser, automatically, or predetermined) are removed. This composite imagehighlights the locations of objects and movements within the subjectarea.

A feature is determined to exist if a sufficient number of sequentialpoints on a circle at a fixed distance meet a minimum thresholdcriteria. For example, if the minimum number of sequential points isdetermined to be 16 and the match requirement is a value greater than10, then a minimum of 16 points in a row on the circle (calculated basedon a variable or fixed distance) must have values greater than 10. Ifthis condition is satisfied, then the center point of the test is deemedto be a feature.

FIG. 31 shows feature tests performed in two dimensions on two differentpoints. Using a minimum number of sequential points of 12, the point onthe left (center of the left circle) does not pass. There are fewer than12 points sequentially on the circle that contain a non-white point. Thepoint on the right (center of the right circle) does pass. There are 13points that are sequentially on the circle surrounding the point.

Feature matching can be done in three dimensions using either planes ora sphere. In the case of a plane, the circle as noted above iscalculated on three different planes. The X-Y plane, the X-Z plane andthe Y-Z plane. If the feature meets the criteria for all planes, then amatch is determined to exist.

FIG. 32 shows a three-dimensional model of the feature test. The planarcircles shown as rings around the outside of the circle represent thecircle used on each axis to determine whether the feature is valid. Atest is done in each plane—XY, XZ and YZ—and if the feature test issuccessful in all three planes, then a feature at the center—the blackdot at the origin in the figure—is determined to be valid.

The location of the target is stored as both 2D coordinate data forimmediate viewing, and 3D coordinate data for reference to movement.Using the matched rotation and scale of the target, the target can beaccurately rendered over the matched area in the subject's view. Bystoring the location in three dimensions, the object can quickly betested in subsequent frames to confirm its location as the user andtarget move.

Another embodiment relates to a method for enhancing positional locationin augmented reality using gadolinium markers.

Gadolinium is a material commonly used to enhance contrast in MRimaging. By mixing gadolinium with a carrier, a surface can be coatedprior to an MR scan. This gives a high contrast image of the coatedsurface suitable for use in target detection for immersive environments.

For example, a patient is having an MR scan to look for lesions in thebrain. The gadolinium infused carrier is spread across the patient'sface prior to the MR scan, which creates strong contrast in thepatient's face. The enhanced contrast from the patient's face is used tocreate a digital image of the patient, allowing facial recognition to beused to identify the patient and position a three-dimensional model ofthe MR scan over the patient's head during a later surgery.

In another example, the gadolinium infused carrier is used as a a markerdrawn on the subject, which is visible in the final MR image and can beused for calibration.

Another embodiment is a method of constructing a three-dimensional modelcomprising the steps of determining image separation distance,identifying missing images, aligning source image and constructingmissing image data, and merging the images to form a three-dimensionalmodel.

Images provided in DICOM format contain data indicating the separationdistance between slices. This data is used to determine the number ofslices required. Absent this data, the lesser of the width and heightdimensions of the image are used to determine depth, creating arectangular model. This value can also be overridden or set by userinput to adjust the model to a correct depth.

Missing images are next identified. Automatic identification is done bylooking at several factors, including numbering of the image files,content of the image files and validity of the image files. Image filesin a sequence are often numbered sequentially. The sequence of numbersis analyzed, and any missing numbers in the sequence are flagged asmissing images. The content of images is analyzed, and images missingsufficient data (e.g., An image that is almost or entirely blank) areflagged as missing images. Invalid image files are files that do notopen as a valid image of the type being used. Automatic generation ofthe three-dimensional image treats flagged images as missing.Alternatively, or in conjunction, a user can review and change missingimages, as well as flag additional images as missing.

Images are then aligned between frames where required. An image isdetermined to be out of alignment if the points of the perimeter are outof alignment from both adjacent images. Therefore, if three sequentialimages have perimeters occupying the same region of the image, adjustedfor scale and changes in shape, the images are determined to be aligned.If the image in the center is out of alignment from the adjacent images,the image is adjusted to be in line by comparing features between theimages and aligning them. This alignment uses the full image and notjust the perimeter.

The final model is created by interpolating missing images. The finalnumber of images required is determined, and the number of images thatmust be added between each existing image pair. Multiple passes aretaken to add the required images. In each pass, one image is addedbetween each existing pair by interpolating the data that exists in theimages. Therefore, in a sequence containing 5 images, there will be 9images after one pass. After a second pass, there will be 16 images.This continues until the desired number of images has been met orexceeded.

Another embodiment is a system for tracking and monitoring a position ofa user's body part (e.g., the user's hands) in augmented or virtualreality environments. In some embodiments, the system comprises a set ofat least one sensor attached to a part of the user's body (e.g., theuser's hand or hands), a means for reading the at least one sensor, anda means of tracking the position of the at least one sensor in two-and/or three-dimensional space. In some embodiments, the means forreading and the means for tracking comprise a processor.

For example, at least one sensor coupled to a user's hands may providepositional data from a user's hands. The at least one sensor may beincluded, for example, in a glove or a pair of gloves to detect thepositions and/or orientations of the user's fingers, palms, wrists, etc.The at least one sensor gathers information about the position(s) of theuser's hands (or portions of the user's hands). A processor is coupledto the at least one sensor, either directly or through an interveningcomponent (e.g., an analog-to-digital converter). For example, the atleast one sensor may be coupled to the processor via a wirelessconnection (e.g., Bluetooth, Wi-Fi, infrared, etc.). The at least onesensor provides, to the processor, information identifying the at leastone sensor's location. The processor then processes the information fromthe at least one sensor to determine positional information associatedwith the user's hands (e.g., location, orientation, movement, etc. ofthe fingers, palms, hands, etc.).

FIG. 33 shows a hand in three different positions, with a sensor on eachof the pinky finger and thumb, shown in grey. As the pinky and thumbmove, the sensor locations are updated. At each frame, the locations in(X, Y, Z) coordinates are shown above.

In a specific example, a user wears a glove equipped with at least onesensor, where the at least one sensor provides position information. Forexample, the at least one sensor may be a gyroscopic or accelerationsensor. The at least one sensor is communicatively coupled, eitherdirectly or through an intervening component (e.g., an analog-to-digitalconverter) to a processor. The system also includes an immersion devicethat either includes or is coupled to the processor. One example of animmersion device is a set of augmented reality glasses.

In operation, the at least one sensor provides to the processorinformation identifying the position of the at least one sensor. Theprocessor then processes the information from the at least one sensorand provides data or instructions to the immersion device. The immersiondevice renders an immersive environment that includes the position ofthe user's hand based on the information obtained from the at least onesensor. Thus, through the immersion device, the user is able to see theposition of his or her hand (or hands) overlaid over the gloves. As theat least one sensor continues to provide positional information to theprocessor, the immersion device is able to render the user's movementsor gestures while wearing the glove in the immersive environment. Thus,the user's movements and/or gestures made wearing the glove allow theuser to interact with augmented reality objects presented by theimmersion device. The augmented reality objects may include, forexample, menus. Pre-determined gestures may be used to indicateselection, and the position of the hand may be used to determine whichitem is being selected.

In some embodiments, the immersive environment rendered by the immersiondevice includes a virtual keyboard, and at least one sensor coupled to auser's hand(s) provides positional information to a processor. Based onthe positional information, the processor identifies at least one key onthe virtual keyboard that corresponds to a position of the user'sfinger. The user can then select keys on the keyboard (e.g., letters,numbers, commands, etc.) using the virtual keyboard in conjunction withthe at least one sensor.

In some embodiments, the immersive environment rendered by the immersiondevice includes a virtual computer interface used for, e.g., gaming,productivity, or media consumption, and at least one sensor coupled to auser's hand(s) (e.g., included in a glove) provides positionalinformation to a processor. Based on the positional information, theprocessor identifies a selection on the virtual computer interface. Theuser can then interact with the virtual computer interface using the atleast one sensor.

Another embodiment relates to a wearable apparatus for full body sensingand feedback comprised of a means for measuring and tracking thewearer's movement, a means for simulating touch senses, a means forsensing objects and surfaces, a means for simulating temperature senses,and a means for restricting user movement.

Sensors and/or actuators as described above can be used to line theinside and/or outside of a full or partial body suit to be worn by auser. Positional sensors as described above can be used to track keypoints of a user's anatomy in order to determine pose, orientation,position, and/or other metrics.

The sensors can be used to detect touches of surfaces as describedabove. These sensors can optionally be used to transmit real-timesensory data via the actuators. These sensors can also be used to recordsensory data as described above.

The wearable device can also include temperature simulators that cantransmit to the wearer recorded sensory data from previous interactionsas described above.

The wearable device can also be connected to an augmented reality orvirtual reality device and operate in conjunction with the device tosimulate objects in that space.

The wearable device can also be connected to a hard drive or otherstorage medium, either embedded within the device or connectedexternally. The device can be used to store software, media or otherinformation to be displayed in either an AR/VR context or via sensorreplay.

Sensors embedded in the wearable device transmit a signal to a receiver,updating the computer as to the location and position of the sensor atany given time. The data is used to determine a user's position andlocation as well as relative locations of limbs and other key points ofanatomy.

Actuators at the joints of the wearable device can also be used torestrict the movement of a user's limbs or other body parts. By usingthese actuators in conjunction with virtual or augmented reality, anadditional level of realism can be brought to the immersive environmentsimulation. For example, if a user is touching a virtual table, theactuators could be used to prevent the user from moving their limb insuch a way that it passes through the table. The actuators can also beused in conjunction with the movement actuators to provide feedback withincreasing pressure as a user pushes against a virtual object.

FIGS. 34A-34C illustrate systems 1000A, 1000B, and 1000C for renderingan immersive environment in accordance with some embodiments. Theimmersive environment may be, for example, an augmented-reality,virtual-reality, enhanced-reality, or immersive-reality environment. Forexample, the immersive environment may be a clinical environment (e.g.,a virtual surgical environment or a therapeutic environment), a gamingenvironment, or a learning environment. The immersive environment mayinclude a virtual peripheral (e.g., a keyboard, menu, mouse, etc.) toenable a user to indicate a selection or provide an input. The immersiveenvironment may comprise a peripheral enabling the user to communicateover a network (e.g., the Internet) to, for example, read, send, orreceive e-mail, conduct a chat session, place a phone call, or engage inpeer-to-peer communication.

The system 1000A, 1000B, 1000C includes at least one electronic device1002 that is configured to be coupled to a body part of a user, wherethe body part may be, for example, a hand, a head, a neck, an arm, aleg, a foot, an eye, a mouth, a facial feature, or a majority or all ofthe user's body. As illustrated in FIGS. 34A-34C, the at least oneelectronic device 1002 includes a sensor 1004 (FIG. 34A), an actuator1006 (FIG. 34B), or both a sensor 1004 and an actuator 1006 (FIG. 34C).In some embodiments, the at least one electronic device 1002 is ascalpel, pair of glasses, a mask, a hat, or a headgear (e.g., helmet).In some embodiments, the at least one electronic device 1002 is an itemattached to or worn by the user. For example, in some embodiments, thesensor 1004 and/or actuator 1006 is attached to or embedded within anitem of clothing, such as, for example, a body suit (e.g., covering aportion, substantially all, or all of the user's body), a sleeve (e.g.,for an arm or leg), a glove, or footwear. The at least one electronicdevice 1002 may include components in addition to the sensor 1004 and/oractuator 1006, or it may simply be the sensor 1004 and/or the actuator1006. The term “electronic device 1002” includes apparatuses that areelectronic devices solely because they include or have attached to themthe sensor 1004 and/or actuator 1006. Therefore, for example, the itemsof clothing and apparel listed above are electronic devices 1002.

The system further includes a processor 1020 that is able to communicateover a communication link 1010A with the at least one electronic device1002. The communication link 1010A may be a wired (e.g., USB, Ethernet,etc.) or wireless (e.g., Bluetooth, Wi-Fi, near-field communication,cellular, infrared, etc.) communication link. The communication link1010A may be simply a bus or direct electrical connection. The processor1020 is configured to execute machine-executable instructions that causethe processor 1020 to obtain data from the at least one electronicdevice 1002 (e.g., data originating from a sensor 1004) and/or providedata to the at least one electronic device 1002 (e.g., data for theactuator 1006 or for controlling the actuator 1006).

As shown in FIGS. 34A-34C, the system 1000A, 1000B, 1000C furtherincludes a rendering device 1030 that is able to communicate over acommunication link 1010B with the processor 1020. The communication link1010B may be a wired (e.g., USB, Ethernet, etc.) or wireless (e.g.,Bluetooth, Wi-Fi, near-field communication, cellular, infrared, etc.)communication link. The communication link 1010B may be simply a bus ordirect electrical connection. The rendering device 1030 is configured toreceive rendering information from the processor 1020 and render theimmersive environment based at least in part on the renderinginformation received from the processor 1020. The rendering device 1030may comprise a display (e.g., an optical projection system, a monitor, ahand-held device, a display system worn on the user's body, etc.). Inembodiments in which the rendering device 1030 comprises a display, thedisplay may be a head-mounted display (e.g., a helmet or harness) thatmay be coupled to the user's forehead. The rendering device may comprisea contact lens, a virtual retinal display, an eye tap, or a hand-helddevice. The rendering device 1030 may comprise a pair of glasses. Insome embodiments in which the rendering device 1030 comprises a pair ofglasses, the pair of glasses comprises a camera that is configured tocapture a real-world view, at least one eye piece, and a projector thatis configured to render the immersive environment by displaying anaugmented or virtual version of the real-world view by projecting animage through or reflected off a surface of the at least one eye piece.In such embodiments, the augmented or virtual version of the real-worldview is based at least in part on the rendering information from theprocessor 1020.

It is to be understood that FIGS. 34A-34C are block diagrams of thesystems 1000A, 1000B, and 1000C. Although the at least one electronicdevice 1002, the processor 1020, and the rendering device 1030 areillustrated separately for convenience of explanation, in animplementation some or all of them may be collocated or combined. Forexample, the at least one electronic device 1002 and/or the renderingdevice 1030 may include the processor 1020. As another example, all ofthe at least one electronic device 1002, the rendering device 1030, andthe processor 1020 may be combined in one apparatus.

In some embodiments, the system does not include an electronic device1002, and a rendering device 1030 includes a sensor 1004, an actuator1006, or both a sensor 1004 and an actuator 1006. FIGS. 35A-35Cillustrate systems 1100A, 1100B, and 1100C in accordance with someembodiments. As illustrated in FIGS. 35A-35C, a processor 1020 is ableto communicate over a communication link 1010B with a rendering device1030. The communication link 1010B may be a wired (e.g., USB, Ethernet,etc.) or wireless (e.g., Bluetooth, Wi-Fi, near-field communication,cellular, infrared, etc.) communication link. The communication link1010B may be simply a bus or direct electrical connection. The processor1020 is configured to execute machine-executable instructions that causethe processor 1020 to obtain data from the rendering device 1030 and/orprovide data to the rendering device 1030 over the communication link1010B. The rendering device 1030 is able to communicate over thecommunication link 1010B with the processor 1020. As explained in thecontext of the embodiments illustrated in FIGS. 34A-34C, the renderingdevice 1030 is configured to receive rendering information from theprocessor 1020, and render the immersive environment based at least inpart on the rendering information received from the processor 1020. Therendering device 1030 may comprise a display (e.g., an opticalprojection system, a monitor, a hand-held device, a display system wornon the user's body, etc.). In embodiments in which the rendering device1030 comprises a display, the display may be a head-mounted display(e.g., a helmet or harness) that may be coupled to the user's forehead.The rendering device 1030 may comprise a pair of glasses. In someembodiments in which the rendering device 1030 comprises a pair ofglasses, the pair of glasses comprises a camera that is configured tocapture a real-world view, at least one eye piece, and a projector thatis configured to render the immersive environment by displaying anaugmented or virtual version of the real-world view by projecting animage through or reflected off a surface of the at least one eye piece.The augmented or virtual version of the real-world view is based atleast in part on the rendering information from the processor 1020. Therendering device may comprise a contact lens, a virtual retinal display,an eye tap, or a hand-held device.

In some embodiments, the system includes memory coupled to or within therendering device 1030, and the rendering device 1030 is configured toobtain additional data from the memory and render the immersiveenvironment based at least in part on the additional data. In someembodiments, the system includes an audio reception device that capturesa sound (e.g., a voice command from the user, an ambient sound from thereal-world environment, etc.) and provides information about the soundto the processor 1020. The rendering information provided by theprocessor 1020 to the rendering device 1030 is them based at least inpart on the information about the sound.

In FIG. 35A, the rendering device 1030 includes a sensor 1004. In FIG.35B, the rendering device includes an actuator 1006, and in FIG. 35C,the rendering device 1030 includes both a sensor 1004 and an actuator1006.

It is to be understood that FIGS. 35A-35C are block diagrams of thesystems 1100A, 1100B, and 1100C. Although the processor 1020 and therendering device 1030 are illustrated separately for convenience ofexplanation, in an implementation they may be collocated or combined.For example, the rendering device 1030 and the processor 1020 may becombined in one apparatus.

In embodiments that include a sensor 1004 (FIGS. 34A, 34C, 35A, and35C), the sensor 1004 may comprise any kind of sensor that can senseinformation about or characteristics of an environment. For example, thesensor 1004 may comprise a piezoelectric sensor, a piezoceramic sensor,a dielectric elastomer sensor, a polyvinylidene fluoride sensor, apiezoresistive sensor, a mechanical sensor, or an electrode. The sensor1004 may comprise a heart rate sensor or a pulse sensor. The sensor 1004may be attached to or embedded within a glove, partial or full bodysuit, scalpel, footwear, a pair of glasses, a mask, a headgear, or theuser's face.

In some embodiments, the sensor 1004 is a positional sensor to detect aposition of the body part of the user. In such embodiments, theprocessor 1020 obtains data from the at least one electronic device1002, which is configured to provide the data to the processor 1020. Thedata represents a position of the body part of the user detected by thesensor 1004. For example, the data may represent an orientation of thebody part (e.g., a hand, a finger, an eye, a head, etc.) relative to areal or virtual object (e.g., a patient's body part, an object in agame, etc.). As just a few examples, the data may represent anorientation of the body part relative to a real or virtual peripheral(e.g., a keyboard, a menu, a mouse, etc.). The real or virtualperipheral may enable the user to communicate over a network (e.g., theInternet, a broadband network (e.g., DSL, cable, fiber), or a localnetwork (Wi-Fi, Ethernet, etc.)) for a purpose such as, for example, toaccess e-mail, conduct a chat session, place a phone call (POTS orVOIP), or engage in peer-to-peer communication. As another example, thedata may represent an orientation of the body part relative to a virtualuser interface. As another example, the data may represent anorientation of the body part relative to a physical object (e.g., a bodypart of a patient). As another example, the data may represent anorientation of the body part relative to a virtual object (e.g., anobject in a game) in the immersive environment. The renderinginformation is based at least in part on the data provided by the atleast one electronic device 1002 to the processor 1020, which indicatesthe position of the body part of the user.

In some embodiments that include a sensor 1004, the user is a patient,and the sensor 1004 is ingestible or injectable into the patient. Thesensor 1004, which may be, for example, a macro-, micro-, ornano-sensor, provides the data to the processor 1020, and the renderinginformation comprises information representing a path of the sensor 1004through the patient. The sensor 1004 may provide the data to theprocessor 1020 using a radio-frequency, Bluetooth, or Wi-Fi signal. Theimmersive environment rendered by the rendering device 1030 includes athree-dimensional view of the patient overlaid by a virtual image of thepath of the sensor within the patient.

In some embodiments that include a sensor 1004, the user is a patient,and the sensor is configured to detect the patient's heartbeat andprovide the data to the processor 1020. The processor 1020 obtains, frommemory, information representing an aspect of a reference heart signaland determines the rendering information based at least in part on thepatient's heartbeat and the information representing the aspect of thereference heart signal. Optionally, the processor 1020 may compare atleast an aspect of the patient's heartbeat to the aspect of thereference heart signal and provide the result of the comparison to therendering device 1030 in the rendering information. The sensor 1004 mayinclude an indicator (e.g., a light source, a display, a speaker, etc.)to indicate a level of the patient's heartbeat. The sensor may include amicrophone. The system may further comprise an electrocardiograph 1060coupled to the processor 1020, and the processor 1020 may obtain asignal generated by the electrocardiograph 1060 and compare at least anaspect of the signal generated by the electrocardiograph to theinformation representing the aspect of the reference heart signal.

In other embodiments that include a sensor 1004, the sensor 1004 is atactile sensor. In such embodiments, the processor 1020 obtains datafrom the at least one electronic device 1002, which is configured toprovide the data to the processor 1020. The data represents acharacteristic of an object in contact with the sensor 1004. Forexample, the characteristic may be a texture, a resistance, atemperature, a hardness, a pressure, a density, a coefficient offriction, or a viscosity of the object. The object may be, for example,a body part of a patient, such as, for example, a prostate, breast,gland, skin, lymph node, abdomen, liver, appendix, gall bladder, spleen,testicle, cervix, knee, head, or shoulder. The rendering information isbased at least in part on the data provided by the at least oneelectronic device 1002 to the processor 1020.

In other embodiments that include a sensor 1004, the sensor 1004 is ahaptic sensor. In such embodiments, the processor 1020 obtains data fromthe at least one electronic device 1002, which is configured to providethe data to the processor 1020. The data represents a characteristic ofan object in contact with the sensor 1004. For example, thecharacteristic may be a texture, a resistance, a temperature, ahardness, a pressure, a density, a coefficient of friction, or aviscosity of the object. The object may be, for example, a body part ofa patient, such as a prostate, breast, gland, skin, lymph node, abdomen,liver, appendix, gall bladder, spleen, testicle, cervix, knee, head, orshoulder. The rendering information is based at least in part on thedata provided by the at least one electronic device 1002 to theprocessor 1020.

In other embodiments that include a sensor 1004, the sensor 1004 is aforce sensor. In such embodiments, the processor 1020 obtains data fromthe at least one electronic device 1002, which is configured to providethe data to the processor 1020. The data represents a force applied tothe sensor 1004 by an object in contact with the sensor 1004. The objectmay be, for example, a body part of a patient, such as a prostate, ahand, a foot, etc., or an object in a game. The rendering information isbased at least in part on the data provided by the at least oneelectronic device 1002 to the processor 1020.

In other embodiments that include a sensor 1004, the sensor 1004 is agyroscopic sensor or an acceleration-detecting sensor. In suchembodiments, the processor 1020 obtains data from the at least oneelectronic device 1002, which is configured to provide the data to theprocessor 1020. The data represents a change in the position of the bodypart of the user detected by the sensor 1004. For example, the data mayrepresent a change in the orientation of the body part (e.g., movementof a hand, a finger, an eye, a head, etc.) relative to a real or virtualobject. As just a few examples, the data may represent a change in theorientation of the body part relative to a real or virtual peripheral(e.g., keyboard, mouse, etc.), a real or virtual menu, a real or virtualpatient's body part, or a real or virtual object in a game. Therendering information is based at least in part on the data provided bythe at least one electronic device 1002 to the processor 1020, whichindicates the change in the position of the body part of the user.

In some embodiments that include a sensor 1004, the sensor 1004 is atemperature sensor. In some such embodiments, the at least oneelectronic device 1002 is an item of clothing comprising a temperaturesensor 1004 for a user who has a limb without feeling. The sensor 1004detects a temperature (e.g., of an object in contact with or near thesensor 1004), and either the sensor 1004 or another component of the atleast one electronic device 1002 (e.g., a speaker, a transmitter, etc.)generates a signal to notify the user that the object may be causingdamage. As a specific example, the at least one electronic device 1002may be a glove comprising a sensor 1004, where the sensor 1004 is atemperature sensor. When the user touches the burner on a stove, thesensor 1004 or another component of the at least one electronic device1002 generates a signal (e.g., an electronic signal, an audible signal,etc.) to indicate that the temperature of the burner exceeds a maximumtemperature. The signal may be in the form of an alert to the user(e.g., an audible sound, a visual indicator, etc.), or it may be anelectronic signal received by an actuator 1006 that removes the user'shand from the burner. As another example, the at least one electronicdevice 1002 may be footwear comprising a sensor 1004, where the sensor1004 is a temperature sensor. If the user's feet are exposed to extremecold, which could potentially cause frostbite, the sensor 1004 oranother component of the at least one electronic device 1002 generates asignal (e.g., an electronic signal, an audible signal, etc.) to indicatethat the temperature of the burner is below a minimum temperature. Thesignal may be in the form of an alert to the user (e.g., an audiblesound, a visual indicator, etc.), or it may be an electronic signalreceived by an actuator 1006 that induces movement (e.g., of the user'sfeet).

In some embodiments that include a sensor 1004, the processor 1020obtains data (e.g., a command from the user, a characteristic (e.g.,texture, resistance, temperature, hardness, pressure, a density,coefficient of friction, viscosity, etc.) of an object (e.g., physicalobject (e.g., patient's body part) or virtual object (e.g., in a game))sensed by the user's body part), from the at least one electronic device1002, which is configured to provide the data to the processor 1020. Theprocessor 1020 sends an instruction to the rendering device, where theinstruction is based at least in part on the data provided by the atleast one electronic device 1002. The rendering device 1030 renders theimmersive environment based at least in part on the instruction.

In some embodiments, the system compares a characteristic of an objectin contact with a sensor 1004 with stored information and optionallyprovides information to the system user regarding whether the object incontact with the sensor 1004 matches or is consistent with the storedinformation. The stored information may have been generated as a resultof the same sensor 1004 having previously been in contact with the sameobject, or it may have been generated as a result of the same sensor1004 having previously been in contact with a similar object, or it mayhave been generated as a result of a different sensor 1004 having beenin contact with the same or a similar object. Alternatively, the storedinformation may have been generated in a manner that is not based ondata from any sensor 1004. In some embodiments, the at least oneelectronic device 1002 includes a first sensor 1004, and the at leastone electronic device 1002 is configured to provide data representing acharacteristic of a first object that is in contact with the firstsensor 1004 to the processor 1020. The object may be, for example, abody part of a patient (e.g., a prostate, a breast, a gland, skin, alymph node, an abdomen, a liver, an appendix, a gall bladder, a spleen,a testicle, a cervix, a knee, a head, or a shoulder). The characteristicmay comprise, for example, a texture, a resistance, a temperature, ahardness, a pressure, a density, a coefficient of friction, or aviscosity. The processor 1020 then obtains, from memory, informationrepresenting an aspect of a second object previously in contact with thefirst sensor 1004 or a second sensor 1004 and compares the first objectto the second object based on the data provided by the at least oneelectronic device 1002 and the information representing the aspect ofthe second object previously in contact with the first sensor 1004 orthe second sensor 1004. In some embodiments, the processor 1020 providesto the rendering device 1030, as the rendering information, informationabout a result of the comparison.

In embodiments that include an actuator 1006 (FIGS. 34B, 34C, 35B, and35C), the actuator 1006 may comprise any kind of component that isresponsible for moving or controlling a mechanism or portion of the atleast one electronic device 1002. The actuator 1006 may be attached toor embedded within a glove, partial or full body suit, scalpel,footwear, a pair of glasses, a mask, a headgear, or the user's face. Insome embodiments, the actuator 1006 is embedded in or attached to anarticle worn by a user (e.g., a glove, a sleeve, a body suit, footwear,etc.), and the actuator 1006 restricts or induces (causes) movement ofthe user, or it causes the user to experience a sensation. For example,the actuator 1006 may comprise a hydraulic actuator, a pneumaticactuator, an electric actuator, a thermal actuator, a magnetic actuator,a mechanical actuator, a piezoelectric actuator, a piezoceramicactuator, a dielectric elastomer actuator, a polyvinylidene fluorideactuator, an electrostatic actuator, a microelectromechanical (MEMS)actuator, or a magnetorheological actuator. As another example, theactuator 1006 may comprise a pneumatic channel (e.g., tubing, aninflatable segment, etc.) that can be filled with air (or another gas orfluid) to restrict or induce movement, or to cause the user toexperience heat or cold, or to cool down the user. As another example,the actuator 1006 may comprise wires (e.g., loose wires or using a wireguides) that restrict or induce gross motor movements (e.g., based onlocations of objects in an immersive environment, based on a program orother instruction source, etc.). Such embodiments may be particularlyuseful when the at least one electronic device 1002 comprises an itemworn by the user (e.g., a glove, a sleeve, a body suit, footwear, anexoskeleton, etc.), or the at least one electronic device 1002 is usedfor resistance training, to assist a user in moving (e.g., walking),etc.

In some embodiments that include in actuator 1006, the at least oneelectronic device 1002 is configured to receive data from the processor1020, the processor 1020 is configured to provide the data to the atleast one electronic device 1002, and the actuator 1006 is configured totake an action based on the data. For example, the actuator 1006 mayrestrict or cause movement of the user's body part based at least inpart on the data. As another example, the actuator 1006 may emulate asensation in the user's body part based at least in part on the data. Asanother example, the actuator 1006 may emulate an aspect of an object(e.g., another person's body, a fluid, a malleable object, an objectfrom a game, etc.) touched by the user's body part based at least inpart on the data. The aspect of the object may include, for example, atexture, a resistance, a hardness, a pressure, a density, a coefficientof friction, or a viscosity.

In some embodiments, the at least one electronic device 1002 comprisesboth a sensor 1004 and an actuator 1006 (see, e.g., FIGS. 34C and 35C).The sensor 1004 detects something, and the at least one electronicdevice 1002 (or, in the case of FIG. 35C, the rendering device 1030)provides information about what was detected as first data to theprocessor 1020. For example, as described above, the sensor 1004 maydetect a position, location, or orientation of the user or a portion ofthe user (e.g., relative to a real or virtual object), or it may detecta change in the position, location, or orientation of the user or aportion of the user (e.g., relative to a real or virtual object), or itmay detect a characteristic (e.g., a texture, resistance, temperature,hardness, pressure, density, coefficient of friction, viscosity, etc.)of or force applied by an object (e.g., a physical object (e.g., a bodypart of a patient), a virtual object (e.g., a virtual object in animmersive environment such as a game), another person (e.g., patient), afluid, etc.) in contact with the sensor 1004. The processor 1020 obtainsthe first data from the sensor 1004 and provides second data to theactuator 1006. The actuator 1006 is configured to (a) cause or restrictmovement of the user's body part based at least in part on the seconddata, (b) emulate a sensation in the user's body part based at least inpart on the second data, or (c) emulate an aspect of the object incontact with the sensor based at least in part on the second data. Therendering device 1030 then renders the immersive environment using therendering information provided by the processor 1020, where therendering information is based at least in part on the first data, thesecond data, or both.

As an example of a system 1000C in which the at least one electronicdevice 1002 includes both a sensor 1004 and an actuator 1006, or asystem 1100C in which the rendering device 1030 includes both a sensor1004 and an actuator 1006, the sensor 1004 and actuator 1006 may beattached to or embedded in an article (e.g., glove, sleeve, partial orfull body suit, footwear, mask, helmet, headgear, pair of glasses, etc.)that is used, for example, in a gaming environment. Assume that the gameinvolves the user walking or wading through a virtual stream, and thesensor 1004 is a positional sensor. The rendering device 1030 renders animmersive environment that is based in part on the user's positionrelative to the virtual stream, rendering the perceived position of thestream based at least in part on the user's movement. The sensor 1004detects the user's position and sends, to the processor 1020, as thefirst data, information about the user's position. As the userapproaches the virtual stream, the rendering device 1030 reflects theuser's progress toward the virtual stream based at least in part on therendering information received from the processor 1020. Using the firstdata provided by the sensor 1004, the processor 1020 monitors the user'sposition, and, when the user's position coincides with the user“entering” the virtual stream, the processor 1020 provides the seconddata to the actuator 1006, which then acts to move or restrict theuser's movement based on the fact that the user's position coincideswith the user entering the virtual stream. For example, the actuator1006 may be embedded in or attached to a sleeve, sock, or portion of abody suit around the user's leg. The actuator 1006 then causes orrestricts movement of the user's legs (for example), and/or emulates thesensation of water flowing over or around the user's legs (for example),and/or emulates the sensation of (for example) the user stepping on arock or coming in contact with a plant or fish in the virtual stream,and/or causes the user to feel a sensation of (for example) warm or coldto emulate the temperature of the stream.

In some embodiments, the at least one electronic device 1002 comprisesan article (e.g., glove, sleeve, full or partial body suit, footwear,etc.) worn by the user, and the sensor 1004 and/or actuator 1006 isattached to or embedded within the article. For example, the article maybe a body suit that it a medical/therapeutic suit, a gaming suit (e.g.,for use in game playing), or a workout suit (e.g., for use in exercise).In some embodiments including a body suit, the body suit compriseselements (e.g., rods, panels, etc.) to give the body suit a more rigidform. For example, the body suit may comprise an exoskeleton (e.g., awearable mobile machine that is powered by a system of electric motors,pneumatics, levers, hydraulics, or a combination of technologies thatallow for or restrict limb movement). Such embodiments may beparticularly useful to assist the user in lifting heavy objects, forstrength training, to assist in walking, etc.

In some embodiments in which the at least one electronic device 1002comprises an article (e.g., glove, sleeve, footwear, suit, etc.) worn bya user, the article comprises a mesh that is selectively used to flattenareas of the suit. Such mesh may comprise, for example, wovenpiezoelectric fibers, pneumatic tubing, hydraulic tubing, or any othermaterial that becomes rigid and flattens when desired. Such embodimentsmay be particularly useful, for example, to allow sensing of a flatsurface with a consistent depth. In embodiments in which an article wornby a user comprises a mesh, an actuator 1006 may optionally be held inthe mesh to maintain relative positioning and allow for an even surfacefeel.

In some embodiments in which the at least one electronic device 1002comprises an article worn by a user, the article comprises pneumatictubing capable of providing cooling within the article worn by the user.Such embodiments allow the user to experience cold sensations fromobjects in an immersive environment and provide cooling if the userbecomes too warm while wearing the article or heating if the userbecomes too cold while wearing the article.

Multiple applications exist for embodiments that include articles wornby the user. One such use, for example, is in massage therapy. Byrecording the sensations of a massage, the massage could then be playedback at a later time allowing the user to receive a massage using onlythe program and wearable device. Another application is using electricalimpulses and timing neurological responses for screening out fakedinjuries as well as timing neural pathways. Another application is inmobility for paraplegic persons. The article worn by the user could, forexample, stimulate the spinal nerves using electric impulses allowing aperson to walk. Alternately, using voice or another means of control,the wearable device could use actuators and sensors to move the limbs ofa person, in effect walking for them.

As another example, the device combined with a display such as virtualor augmented reality, can be used as a personal computer, including butnot limited to uses for gaming, productivity and media.

In another example, the device can be used in training students insurgical procedures. The device transmits the resistance and othersensations present during a surgical procedure in conjunction with avirtual or augmented reality environment to give the student a realisticexperience of performing the procedure.

In another example, the device can be used in training students toperform a physical examination of a patient. A virtual patient is seenin either virtual or augmented reality, and the sensations of touchingthe patient are transmitted through the device. This allows for thestudent to be trained in the methodology of performing a physicalexamination, including diagnosis of conditions that can be determinedthrough sensory feedback.

In another example, training of coordination and muscle strengtheningfor patients with conditions such as dysgraphia is done by combiningaugmented reality with the apparatus. A ball is shown in augmentedreality, and the sensations of the ball are transmitted to the userthrough the apparatus. Resistance is felt through the actuators,allowing the user to practice movement and build muscle strength andcontrol.

In another example, an augmented reality gym is displayed to the user.The actuators simulate the feeling of exercise apparatus and resistancefor different weights. This allows the user to do virtual weightexercises without the risk of being injured by the exercise equipment.

In another example, patients with multiple sclerosis, arthritis or otherpermanent disabilities can be assisted and trained in day to dayactivities, either through augmented reality or performance of realtasks, with muscular and motor control assistance from the apparatus.The amount of assistance provided can be varied and reduced to allow theuser to learn to adjust to their disability without assistance.

In another example, an amputee patient can be assisted by the device inperforming day to day tasks, with the assistance being gradually reducedover time to allow the patient to learn to perform their day to dayactivities without assistance. This can be done either in real worldinteractions, or in an immersive environment.

In another example, the wearable device can be used for rehabilitativefunctions, such as assisting paraplegic patients in walking to avoidblood clotting, atrophy and other forms of damage due to inactivity.

Another embodiment is an augmented reality and virtual reality game, inconjunction with the above device, used for assistance in learning tofunction with dyspraxia. A variety of different activities designed toimprove fine motor skills, gross motor skills and motor planningcomprise the game. Gross motor skills can be improved, for example, witha simulation of crawling through virtual tunnels. Fine motor skills canbe improved, for example, by completing a virtual puzzle withprogressively smaller pieces that must be manipulated by hand into thecorrect positions. Motor planning can be improved by combining motortasks with mental tasks, such as walking while counting by two's, orcatching virtual butterflies while counting them out loud.

Muscle strength can also be trained using the game in conjunction withthe disclosed device for actuation and sensing. By creating virtualactivities such as rolling a progressively heavier ball up a hill, withactuarial simulation of the resistance of both climbing the hill andpushing the ball, muscle strength can be increased over time.

Another embodiment relates to a method of using sensors to record andreplay sensory data to simulate the feel of real world objects. Thesensors can be, but are not limited to, haptic sensors, piezoelectricsensors, piezoceramic sensors, dielectric elastomer sensors,polyvinylidene fluoride, and/or piezoresistive sensors. For example, asurgeon could touch a human limb while the sensors record theresistance. The sensor readings can then later be replayed to emulatethe feeling of touching the recorded limb. A variety of readings on thesame type of subject matter can be used to develop a profile forparticular subjects. The profile can then be used to determine whether asurface touched matches an existing profile. This allows foridentification of the surface.

The sensors take a reading based on pressure applied. The methods fortaking these readings are commonly known among those skilled in the art.Grouping many sensors together in a surface, such as the surface of aglove, allows for a large area to be sensed at once time. By reading theresults of these sensors, a profile can be generated for a surfacetouched by the sensors. Using multiple readings as the sensors moveacross the surface can give enhanced readings. Differences in pressurebetween neighbouring sensors assists in determining texture of thesurface. Measuring the difference in pressure between sensors assists indetermining the surface tension of the surface being touched.Additionally, temperature readings can be taken from the surface andrecorded to enhance the recording accuracy.

FIG. 36 shows a series of haptic sensors near/touching a surface. Someof the haptic sensors are touching the surface, and therefore receivingnon-baseline readings. The readings from all non-baseline sensors arerecorded.

Using recorded surface data, actuators touching a user can be used toreplicate the feel of the original surface. The actuators can be, butare not limited to, magnetic actuators, pneumatic actuators,piezoelectric actuators, piezoceramic actuators, dielectric elastomeractuators, polyvinylidene fluoride actuators, electrostatic actuators,and/or magnetorheological actuator. The actuators exert pressure in thepattern recognized in the recording, and adjust as a user moves theirhands over a virtual surface. This transmits both texture and surfacetension data, allowing for a realistic feel. Temperature can also bereplayed from the recording, allowing for a more realistic feeling.

FIG. 37 shows a series of actuators being touched by a user's finger.The actuators being touched are partially actuated to simulate apreviously recorded surface sensation.

The sensors can also be used to identify a surface based on a databaseof recordings. A surface touched is compared to existing recordings todetermine whether the profile matches, and if a sufficiently strongmatch is encountered, the surface can be deemed to be a match.

For example, during an examination of a patient, the sensor devicerecords the feeling of the patient's abdomen. In a subsequentexamination, the results of previous sensor readings can be compared todetermine if a change has occurred. If a change has occurred, it canthen be analyzed and either a diagnosis can be attempted, or the usercan be notified that a change has occurred.

Another embodiment relates to a method for using recorded sensory datafor training for home and professional diagnostic medicine. For example,a sensory recording of a normal vs. inflamed prostate could be used toteach a doctor to identify the difference in a training environment.

Using sensory data recorded with the aforementioned sensor data, a usercan be trained to identify the difference between different types ofsurfaces in an immersive environment.

For example, a doctor could be trained to identify a prostate withissues by examining a patient in virtual reality using sensor replayfeedback.

XIX. Diagnostic/Injury Analysis and Confirmation

Another embodiment relates to a method and apparatus for analyzinginjuries and other medical conditions for the purpose of insuranceadjusting. Insurance adjusters are often tasked with determining whethera claimant's injury is real or exaggerated. By analyzing a patient usingmachine-learning algorithms, it can be determined whether a claimant'sresponse to an injury remains the same or varies over time. For example,if a claimant says that they have a knee injury, monitoring their gaitas they walk can reveal whether the alteration in their walk isconsistent, giving the adjuster a better idea as to the validity of theclaim.

Another embodiment relates to a method and apparatus for performingoptometric exams using augmented reality, virtual reality or otherimmersive environments. Optometrists diagnose and determine eyeconditions and prescription requirements using a series of tests withA-B comparisons. These tests can be administered in an immersiveenvironment by displaying the tests to the user, and accepting verbal,gesture or other user input responses to make a diagnosis.

Another embodiment relates to a method for analyzing injuries and othersuch medical conditions for the purpose of insurance adjusting, bycomparing symptoms for consistency to ensure validity of claims. When asubject is examined for a subjective injury, the decision as to whetheror not an injury is real is made primarily subjectively. By usingartificial intelligence (for example machine learning) to analyze video,a subject can be analyzed to determine if differences in the responsesto injury amount to a real injury or not. For example, if a subject isclaiming compensation for an injured knee, the software is able todetermine whether the subject's gait is consistent. While an uninjuredpatient may be able to convince a person that the injury is real, thesoftware can analyze the motions involved and determine whether they areconsistent. Inconsistencies in the response to the injury are a strongindicator that the injury is exaggerated or fake.

FIG. 38 shows a sequence of a subject walking. By analyzing the anglesof the body, such as the knees, hips and ankles, a profile of thesubject's walk can be established. By comparing this profile to asubsequently viewed or recorded walk, analysis can be performed todetermine if the gait matches the original recording.

Another embodiment relates to a method for doing optometric exams usingaugmented or virtual reality. Optometric exams are done by asking thepatient a series of questions to identify which view is better for thepatient. By using an immersive environment to emulate the effects ofdifferent prescriptions, a user can interactively select whichprescriptions are better for their vision. By responding to differentsets of prescriptions, optimal vision can be provided and a prescriptiondetermined for the user.

FIG. 39A shows an image that is blurry, the first of two images shown toa user when determining what prescription is better for them. FIG. 39Bshows the same image, but not blurry. In this case, the user shouldselect FIG. 39B to indicate that this gives a clearer image.

For example, a user wearing a set of augmented reality glasses can beginan optometric exam in augmented reality. The exam begins by asking theuser to read a standard optometric chart. The chart is presented inaugmented reality at a standard distance in an immersive environment.The user reads the letters from the requested line of the chart, and theresponses are analyzed for accuracy. If the user is unable to correctlyread the letters of the chart, then comparison testing is initiated.Emulations of different lens strengths are shown to the user in pairs,with the response to each pair being used to narrow the prescription ofthe user. When the optimal prescription strength has been determined,the user is presented the chart again, this time filtered by thedetermined prescription. If the user passes the eye exam at this point,the prescription is determined to be valid and can be presented to theuser. The prescription could also be printed or saved.

XX. Surgical Procedure Recording/Playback

Another embodiment relates to a method for recording of medical, dental,or surgical procedures or clinical visits for later playback. Suchrecordings may be useful, for example, for purposes such as teaching,patient comfort (e.g., being able to view his or her own surgery afterthe fact to verify that nothing unexpected happened while the patientwas under anaesthetic), distance learning/observation, problem tracking(e.g., to determine whether anything occurred during a procedure orvisit that could explain an unexpected ailment or pain), etc. Therecording creates a record of a procedure or visit and may be ofinterest to insurance companies, patients, doctors/clinicians/dentistsgiving a second opinion, students, etc.

Recording of a medical, dental, or surgical procedure or clinical visitusing a wearable device including virtual components allows for laterplayback. This recording can be used for, but is not limited to,evidence in legal actions (e.g., malpractice or wrongful deathlawsuits), training of students, review of procedures, and/or audits.

The recording is done by conventional video recording, coupled with arecording of the virtual objects and their positions during the courseof the recording. Audio recording may also be performed. Playback isaccomplished by simultaneously playing the video recording with anaugmented overlay showing the virtual objects and positions.

VIII. AR/VR Facilities

Another embodiment relates to a method for sharing data between devicesin an immersive environment in a group environment.

Another embodiment relates to a method for conferencing betweenpractitioners and/or users in an immersive environment.

Another embodiment also a method for identifying inflammation and otherhot spots in an immersive environment.

Another embodiment also a method and apparatus for displaying fullfield-of-view images in an augmented reality environment.

Another embodiment also a device for displaying an augmented realityenvironment.

Another embodiment also a method for sharing data between devices in animmersive environment in a group environment.

Another embodiment relates to a method for sharing the display of anaugmented reality environment comprising the steps of encoding data froma camera or other imaging device into a transmissible format,transmission of augmented reality object data, transmission of augmentedreality target locations, scales and orientations, synchronization ofvideo and augmented reality target data and display of a combined videoand augmented reality image. For example, live streaming an augmentedreality view from one user or location to a virtual reality receiver atanother location.

Encoding data from a camera or other imaging device can be accomplishedin a variety of ways by those skilled in the art. In one such example,the MPEG system of compressing video is used. MPEG allows for minimaldata transmission by encoding only the differences between cameraframes. Augmented and virtual reality objects can be composed in avariety of different ways by those skilled in the art. There existseveral standardized formats for storing three-dimensional models, suchas, but not limited to, the 3DO (Three-dimensional object) file. Byconverting virtual objects into a format such as 3DO and optionallycompressing the data using common compression methods, the model datacan be quickly transmitted to other clients connected to the simulation.Textures used to cover the three-dimensional models are also transmittedin a compressed image format. An example of one such format is the PNG,or portable network graphic, format whose algorithm is commonly known.

The transmission of augmented reality data can be accomplished bysending a video frame encoded to include unique identification numbersinforming the client computer which objects are present in the presentscene, their locations, orientations scales. These locations,orientations and scales are all normalized to allow for accuratereproduction in the client display. This normalization accounts fordifferences in resolution, aspect ratio and other aspects of the displaydevice. Synchronization is handled between the objects and the image byproviding the object data with each frame, such that a missed or skippedframe does not cause an error in the visualization. If an object that isnot known to the client is encountered, the client initiates a requestto the host for the object data. The object is omitted until the data isreceived by the client, which happens asynchronously over theconnection.

When the client receives an image frame, with or without thethree-dimensional object data, the display is shown to the client as acomposite image. The image data is displayed on the display device withthe augmented reality objects added in their respective locations.Missing objects are skipped and requests are sent to the host for themissing objects data files. If the files are unavailable, the client isinformed such that repeated requests for invalid objects are not made.

The client can also interact with the objects in augmented or virtualreality, allowing for a shared simulation environment. Interactions aresent over the network from host to clients or from clients to host. Inthe case of conflicting movements, the host is deemed to have priorityand the client interactions will be ignored.

In another embodiment, clients and hosts can have the same objects withdifferent locations, orientations and scales, allowing a client tointeract differently with objects than the host. In these cases, theobjects that are out of synchronization are noted by the client and therequests from the host that affect that object are ignored. Optionally,the host or client can have the option to force synchronization of oneor multiple objects, which will bring back into alignment the locations,orientations and sizes of the objects. This is particularly useful, forexample, in a learning environment where a teacher is showing aparticular virtual object to a class. Each student can then interact ontheir own with the object to examine it, and when the teacher wishes tocontinue the lesson they can force resynchronization of the virtualobject.

Data can be shared between devices and displayed in an immersiveenvironment to users in separate locations. This data is transmitted viaa network or other means of wireless or wired communication. The datacan then be used to share interactions between different locations andusers.

For example, a group of doctors could do a virtual consultation of apatient where one device is present with the subject, and each doctor orgroup of doctors have a device for viewing the immersive environment.Communication and interaction is possible by each user involved in theenvironment.

Another embodiment is a method for conferencing between practitioners inan immersive environment.

Live conferencing can be accomplished between devices by transmittingvideo as well as virtual object definitions and positions. When theconnection is first established, definitions for virtual objectsexisting in the scene are transmitted. Once these definitions have beentransmitted, the objects can be identified with unique identificationnumbers. These identification numbers can be provided combined withpositional, scalar and orientation information to allow for display ofthe object.

Multiple clients can connect to the conferenced immersive environment,and can optionally communicate via audio, text or other known means ofcommunication available over a network. These conferences can be used,for example (but not limited to), live viewing of procedures, assistancefrom another client during a complicated procedure, and teaching byexample.

Another embodiment relates to a method for identifying inflammation andother hot spots in an immersive environment.

By monitoring infrared information, inflammation of joints, soft tissueinjuries and other such injuries can be detected. This information canalso be used to gather data such as heart rate.

For example, using infrared viewing technology, a physiotherapist canlook at a patient and determine where soft tissue injuries exist thatmay need relief or avoidance during treatment.

As another example, a chiropractor could examine a patient using theinfrared view to determine where injuries are and to assist in painrelief and adjustment of spinal alignment.

Another embodiment relates to a method and apparatus for displaying fullfield-of-view images in an augmented reality environment. Augmentedreality images can be displayed across the entire field-of-view (FOV) ofthe user's eye. By displaying an augmented reality image across the fullFOV of the user's eye, a more immersive and believable augmented realityenvironment can be created. One method for displaying such an immersiveaugmented reality environment is to use a transparent LCD or LED displaymodule, curved to fill the user's FOV. Another method for providing suchan environment is to use retinal projection. By using a photon source,such as (but not limited to) a laser diode, images are projecteddirectly onto the retina of the user. Optionally, by tracking theposition of the pupil, the photon source can be manoeuvred to remaincoplanar to the pupil, and therefore to the retina.

The required resolution to display can also be reduced by usingproperties of the human eye. In the human eye, a small area called thefovea exists wherein the eye has only receptor cones, and no rods. Thisarea of the eye is responsible for providing fine detail to the centerof vision. By ensuring that the fovea receives high-resolution imagery,the remaining area of the eye can be provided with lower resolutioninformation that will be undetectable to the eye, but will allow formuch faster rendering of the subject image. The human eye has anapproximate FOV of 160 degrees in width and 135 degrees in height. Thefovea accounts for approximately 1-2 degrees in width and height at thecenter of the eye. This region is capable of seeing much greaterresolution than the surrounding eye, with acuity being reduced thefurther from the fovea the image is presented.

Augmented reality can also be presented using curved lenses. By adheringor affixing in place one or more curved panes, a display can bepresented on while still allowing viewing through the panes. Layers canbe stacked with each layer representing a set of points. Transparentprisms or other transparent means of changing the angle of light passingthrough the medium are used to alter the path of light projected intothe layer. The prisms or other means of changing the angle of light arestaggered such that one can be targeted without passing through anyother.

Another embodiment (FIG. 40) relates to a device for an augmentedreality display. The augmented reality device is composed of a displaydevice, such as outlined above, an audio capture device, a monaural orstereo audio output device, a CPU, a GPU, and one or more highdefinition cameras. The embodiment can also optionally include a LiDARsystem. The embodiment can also optionally include infrared sensors,including but not limited to FLIR.

In one embodiment, the device uses a single high definition camera tocapture the environment.

In another embodiment, the device uses two high definition cameras. Onefaces the user to track their eye movements and positions, while theother is used to capture the environment.

In another embodiment, the device uses two high definition cameras tocapture the environment in a binocular fashion.

In another embodiment, the device uses three high definition cameras,with one camera facing the user to track eye movements and positions,while the other two cameras are used to capture the environment in abinocular fashion.

In another embodiment, the device uses three high definition cameras,with two cameras facing the user, and one camera used to capture theenvironment. The cameras facing the user are each used to track eyemovements and positions of a single eye.

In another embodiment, the device uses four high definition cameras,with two cameras facing the user and two cameras used to capture theenvironment in a binocular fashion. The two cameras facing the user areeach used to track eye movements and positions of a single eye. Thecameras facing externally and internally are synchronize to ensure thatthe view is consistent.

Any of the various methodologies disclosed herein and/or user interfacesfor configuring and managing the disclosed apparatuses and systems maybe implemented by machine execution of one or more sequences ofinstructions (including related data necessary for proper instructionexecution). Such instructions may be recorded on one or morecomputer-readable media for later retrieval and execution within one ormore processors of a special purpose or general-purpose computer systemor consumer electronic device or appliance, such as the various systemcomponents, devices and appliances described above (e.g., programmedprocessor(s) 1020 as shown in several of the drawings herein).Computer-readable media in which such instructions and data may beembodied include, but are not limited to, non-volatile storage media invarious non-transitory forms (e.g., optical, magnetic or semiconductorstorage media) and carrier waves that may be used to transfer suchinstructions and data through wireless, optical, or wired signallingmedia or any combination thereof. Examples of transfers of suchinstructions and data by carrier waves include, but are not limited to,transfers (uploads, downloads, e-mail, etc.) over the Internet and/orother computer networks via one or more data transfer protocols (e.g.,HTTP, FTP, SMTP, etc.).

In the foregoing description and in the accompanying drawings, specificterminology and drawing symbols have been set forth to provide athorough understanding of the disclosed embodiments. In some instances,the terminology and symbols may imply specific details that are notrequired to practice those embodiments. For example, any of the specificdimensions, numbers of components (cameras, projections, sensors, etc.),component circuits or devices and the like can be different from thosedescribed above in alternative embodiments. Additionally, links or otherinterconnection between system components or functional blocks may beshown as buses or as single signal lines. Each of the buses canalternatively be a single signal line, and each of the single signallines can alternatively be buses. Signals and signalling links, howevershown or described, can be single-ended or differential. The term“coupled” is used herein to express a direct connection as well as aconnection through one or more intervening circuits or structures.Device “programming” can include, for example and without limitation,loading a control value into a register or other storage circuit withinthe device or system component in response to a host instruction (andthus controlling an operational aspect of the device and/or establishinga device configuration) or through a one-time programming operation(e.g., blowing fuses within a configuration circuit during deviceproduction), and/or connecting one or more selected pins or othercontact structures of the device to reference voltage lines (alsoreferred to as strapping) to establish a particular device configurationor operation aspect of the device. The terms “exemplary” and“embodiment” are used to express an example, not a preference orrequirement. Also, the terms “may” and “can” are used interchangeably todenote optional (permissible) subject matter. The absence of either termshould not be construed as meaning that a given feature or technique isrequired.

Various modifications and changes can be made to the embodimentspresented herein without departing from the broader spirit and scope ofthe disclosure. For example, features or aspects of any of theembodiments can be applied in combination with any other of theembodiments or in place of counterpart features or aspects thereof.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

It should be noted that the various circuits disclosed herein may bedescribed using computer aided design tools and expressed (orrepresented), as data and/or instructions embodied in variouscomputer-readable media, in terms of their behavioural, registertransfer, logic component, transistor, layout geometries, and/or othercharacteristics. Formats of files and other objects in which suchcircuit expressions may be implemented include, but are not limited to,formats supporting behavioral languages such as C, Verilog, and VHDL,formats supporting register level description languages like RTL, andformats supporting geometry description languages such as GDSII, GDSIII,GDSIV, CIF, MEBES and any other suitable formats and languages.Computer-readable media in which such formatted data and/or instructionsmay be embodied include, but are not limited to, computer storage mediain various forms (e.g., optical, magnetic or semiconductor storagemedia, whether independently distributed in that manner, or stored “insitu” in an operating system).

When received within a computer system via one or more computer-readablemedia, such data and/or instruction-based expressions of the abovedescribed circuits can be processed by a processing entity (e.g., one ormore processors) within the computer system in conjunction withexecution of one or more other computer programs including, withoutlimitation, net-list generation programs, place and route programs andthe like, to generate a representation or image of a physicalmanifestation of such circuits. Such representation or image canthereafter be used in device fabrication, for example, by enablinggeneration of one or more masks that are used to form various componentsof the circuits in a device fabrication process.

In the foregoing description and in the accompanying drawings, specificterminology and drawing symbols have been set forth to provide athorough understanding of the disclosed embodiments. In some instances,the terminology and symbols may imply specific details that are notrequired to practice those embodiments. For example, any of the specificdimensions, numbers of components (cameras, projections, sensors, etc.),voltages, pixel array sizes, signal path widths, signaling or operatingfrequencies, component circuits or devices and the like can be differentfrom those described above in alternative embodiments. Additionally,links or other interconnection between system components, functionalblocks, integrated circuit devices, or internal circuit elements orblocks may be shown as buses or as single signal lines. Each of thebuses can alternatively be a single signal line, and each of the singlesignal lines can alternatively be buses. Signals and signalling links,however shown or described, can be single-ended or differential. Theterm “coupled” is used herein to express a direct connection as well asa connection through one or more intervening circuits or structures.Device “programming” can include, for example and without limitation,loading a control value into a register or other storage circuit withinthe device or system component in response to a host instruction (andthus controlling an operational aspect of the device and/or establishinga device configuration) or through a one-time programming operation(e.g., blowing fuses within a configuration circuit during deviceproduction), and/or connecting one or more selected pins or othercontact structures of the device to reference voltage lines (alsoreferred to as strapping) to establish a particular device configurationor operation aspect of the device. The term “light” as used to apply toradiation is not limited to visible light, and when used to describesensor function is intended to apply to the wavelength band or bands towhich a particular pixel construction (including any correspondingfilters) is sensitive. The terms “exemplary” and “embodiment” are usedto express an example, not a preference or requirement. Also, the terms“may” and “can” are used interchangeably to denote optional(permissible) subject matter. The absence of either term should not beconstrued as meaning that a given feature or technique is required.

Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation, including meanings implied fromthe specification and drawings and meanings understood by those skilledin the art and/or as defined in dictionaries, treatises, etc. As setforth explicitly herein, some terms may not comport with their ordinaryor customary meanings.

As used in the specification and the appended claims, the singular forms“a,” “an” and “the” do not exclude plural referents unless otherwisespecified. The word “or” is to be interpreted as inclusive unlessotherwise specified. Thus, phrases of the form “A or B” are to beinterpreted as meaning all of the following: “both A and B,” “A but notB,” and “B but not A.” Any use of “and/or” herein does not mean that theword “or” alone connotes exclusivity.

As used in the specification and the appended claims, phrases of theform “at least one of A, B, and C,” “at least one of A, B, or C,” “oneor more of A, B, or C,” and “one or more of A, B, and C” areinterchangeable, and each encompasses all of the following meanings: “Aonly,” “B only,” “C only,” “A and B but not C,” “A and C but not B,” “Band C but not A,” and “all of A, B, and C.”

To the extent that the terms “include(s),” “having,” “has,” “with,” andvariants thereof are used in the detailed description or the claims,such terms are intended to be inclusive in a manner similar to the term“comprising,” i.e., meaning “including but not limited to.”

The drawings are not necessarily to scale, and the dimensions, shapes,and sizes of the features may differ substantially from how they aredepicted in the drawings.

Various modifications and changes can be made to the embodimentspresented herein without departing from the broader spirit and scope ofthe disclosure. For example, features or aspects of any of theembodiments can be applied in combination with any other of theembodiments or in place of counterpart features or aspects thereof.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

The following non-exhaustive methods, systems and system components aredisclosed herein:

A method for providing an augmented or virtual reality surgical overlay,comprised of elements including, but not limited to, heads-up-display(HUD), medical imaging display, vital statistics display, patientinformation display, procedural information and other data.

A method for displaying surgical targets and other pertinent medicaland/or anatomical data in an augmented or virtual reality surgicalenvironment.

A method for providing an augmented or virtual reality surgical overlayfor laparoscopic procedures, comprised of elements including, but notlimited to, mapping of laparoscopic device path, display of laparoscopicdevice position, display of laparoscopic imaging data, and system fortaking notes generally and related to specific points.

A method for providing an augmented or virtual reality anatomicaldisplay, comprised of elements including, but not limited to, anatomicaldiagramming and labelling, veterinary anatomy, and dissectionsimulations.

A method for combining gross anatomy with problem based learning (PBL).

A method for providing an augmented or virtual reality medicalsimulation, comprised of elements including, but not limited to,diagnostic simulations, surgical simulations, procedural simulations,previewing surgeries based on patient imaging, and group simulations forpurposes such as teaching.

A method for providing an augmented or virtual reality view for cosmeticsurgical usage, comprised of elements including, but not limited to,patient previews, verification of results, and assistance duringsurgery.

A method for displaying a heads-up display (HUD) in augmented or virtualreality composed of two or three-dimensional images superimposed on orintegrated into the environment being viewed.

A method for providing an augmented or virtual reality display fororthodontic use, comprised of the ability to display previews oforthodontic work, a method for showing future tooth alignments andpositions, a method of determining shapes and sizes of dental devices,and a method of generating data files of dental devices for 3D printing.

A method for mapping and analyzing human bodies, comprised of scanningof the body, storing of surface data, marking of important features suchas melanoma, moles, rashes, other skin conditions and remarkablefeatures (either automatically or by human interaction).

A method for timing MR imaging based on position of the patient's body,for example using the height of the chest to ensure that images aretaken at the same point during the breathing process to give a morestable image.

A method for analyzing injuries and other such medical conditions forthe purpose of insurance adjusting, by comparing symptoms forconsistency to ensure validity of claims.

A method of using augmented or virtual reality combined with artificialintelligence for the purpose of testing and teaching materials tostudents.

A method of using sensors to record and replay sensory data to simulatethe feel of real world objects. For example, a surgeon could touch ahuman limb while the sensors record the resistance. The sensor readingscan then later be replayed to emulate the feeling of touching therecorded limb. A variety of readings on the same type of subject mattercan be used to develop a profile for particular subjects. The profilecan then be used to determine whether a surface touched matches anexisting profile. This allows for identification of the surface.

A method for using recorded sensory data for training for home andprofessional diagnostic medicine. For example, a sensory recording of anormal vs. inflamed prostate could be used to teach a doctor to identifythe difference in a training environment.

A method for doing optometric exams using augmented or virtual reality.

A method for using augmented reality in laser eye resurfacing (LASIK)surgery.

A method for using augmented or virtual reality for psychologicaldesensitization of phobias, for example fear of spiders could usemonitoring of vital signs to determine the user's level of stress andeither increase or decrease exposure to spiders in an immersiveenvironment to help a user get over a specific phobia.

A method and apparatus for psychological treatment using a virtualperson.

A method for augmented or virtual reality simulation for the purpose oftraining a user in first aid.

A method for doing intelligence quotient testing using augmented orvirtual reality.

A method for assisting psychiatric and psychological patients using areactive augmented or virtual reality.

A method for determining psychosis and phobias in patients using vitalsigns tracking combined with augmented or virtual reality stimuli.

A method for diagnosing trauma victims using augmented or virtualreality combined with vital signs measurements in order to determinesources of potential past or current traumas. For example, if a child isabused, using simulated images that may mimic situations similar tothose experienced by the child and monitoring their vital signs can helpto determine whether trauma is present, particularly in cases where thepatient is unaware or unwilling to discuss the events.

A method for diagnosing epilepsy using EEG or MEG and vitals sensing,light events and other such stimuli in virtual or augmented reality tomonitor responses.

A method for using pupil dilation, eye movement and pupil location fordetermination of response to stimuli, detecting fabrication in stories,and other such vital signs detection.

A method for interpreting raw MRI signal data into compositethree-dimensional models for use in virtual reality, augmented reality,and 3d printing applications.

A method for controlling the visualization of a three-dimensional objectdisplayed in virtual reality, augmented reality, or other immersiveenvironment comprising the steps of determining the requisite change invisualization, and updating the three-dimensional object. An apparatusfor controlling the visualization of a three-dimensional objectdisplayed in virtual reality, augmented reality, or other immersiveenvironment comprising a means of determining the requisite change invisualization, and a means for updating the three-dimensional object.The process may be performed automatically by a system or may be guidedinteractively by an operator. Applications include, but are not limitedto, virtual reality, augmented reality and three-dimensional printing.

A method for visualizing medical imaging data in augmented reality,virtual reality, or other immersive environment, comprising the steps oflocating the subject, determining subject position, determining subjectorientation, and rendering the medical imaging data. An apparatus forvisualizing medical imaging data in augmented reality, virtual reality,or other immersive environment, comprising a means for locating thesubject, a means for determining subject position, a means fordetermining subject orientation, and a means for rendering the medicalimaging data. The process may be performed automatically by a system ormay be guided interactively by an operator. Applications include, butare not limited to, visualization for the purpose of surgicalprocedures, visualization for the purpose of medical testing,visualization for the purpose of surgical training, visualization forthe purpose of medical training, visualization for the purpose ofphysiotherapy, visualization for the purpose of laser surgery, andvisualization for the purpose of physical diagnostics.

A method for enhancing positional location in augmented reality usinggadolinium markers.

A method and apparatus for constructing a three-dimensional modelcomprising the steps of determining image separation distance,identifying missing images, aligning source image and constructingmissing image data, and merging the images to form a three-dimensionalmodel.

An apparatus for tracking and monitoring positions of users' hands inaugmented or virtual reality environments, comprised of a set of sensorsattached to the user's hands, a means for reading the sensors, and ameans of tracking the positions of the sensors in two and/orthree-dimensional space.

A wearable apparatus for full body sensing and feedback comprised of ameans for measuring and tracking the wearer's movement, a means forsimulating touch senses, a means for sensing objects and surfaces, ameans for simulating temperature senses, and a means for restrictinguser movement.

A method for sharing data between devices in an immersive environment ina group environment.

A method for recording of surgical procedures for later playback.

A method for conferencing between practitioners in an immersiveenvironment.

A method for teaching students using augmented or virtual realitycombined with artificial intelligence.

A method and apparatus for displaying full field-of-view images in anaugmented reality environment.

An apparatus for an augmented reality display.

A method of sensing and displaying liposuction data, including but notlimited to, volume of material removed from the patient, and mock-ups ofpost-surgical results.

A method and apparatus for detecting fluid using a hygrometer attachedto a cannula.

A method for tracking a positional sensor ingested by a patient andtracked via augmented or virtual reality overlay.

A method and apparatus for adaptive radiation shielding for radiationtherapy using augmented reality to direct the location and size of theexposure aperture.

A method and apparatus for adaptive radiation shielding comprising amembrane or other container filled with a lead suspension solution.Additional membranes can be added to the apparatus containing solutionssuch as a ferromagnetic solution.

A method for creation and printing of three-dimensional models forprosthetics.

A method and apparatus for magnetic resonance imaging comprised of astandard MRI machine with the RF frequency coil replaced by multiplecoils operated independently or in a synchronized fashion in order togenerate an improved MR image.

A method for using customized RF coils in MR imaging in order to createimages with higher signal to noise ratios and higher contrast.

A method for analyzing MR images with a moving patient for diagnosticpurposes.

A method for interacting with an immersive environment using cerebralactivity monitoring.

A method for magneto-stabilization of patient anatomy.

A method for separating healthy tissue from cancerous tissue.

A method for identifying microscopic skin conditions using a highdefinition camera.

An apparatus for auditory cardiographic analysis.

An apparatus for rapid tracing and interpretation of cardiographicrhythm anomalies.

A method for identifying inflammation and other hot spots in animmersive environment.

A method and apparatus for performing automated or user-diagnosticprocedures.

A method for voice recognition used to translate speech between patientsand practitioners in order to facilitate communication.

A method for analyzing speech in a practitioner and patient environmentto assist in diagnosis and verify plausibility of identified diagnoses.

We claim:
 1. (canceled)
 2. A system for rendering an immersiveenvironment, the system comprising: a wearable device comprising: a meshhaving a selective rigidity, the selective rigidity allowing the mesh totransition between being in a malleable state and being in a rigidstate, and an actuator coupled to the mesh for controlling the selectiverigidity of the mesh; a processor capable of being communicativelycoupled to the actuator; and a rendering device capable of beingcommunicatively coupled to the processor, wherein: the processor isconfigured to execute machine-executable instructions that, whenexecuted by the processor, cause the processor to provide data to theactuator, the data for controlling the selective rigidity of the mesh,and the rendering device is configured to: receive rendering informationfrom the processor, the rendering information identifying a position ororientation of at least a portion of the wearable device, and render theimmersive environment based at least in part on the renderinginformation from the processor.
 3. The system recited in claim 2,wherein the mesh comprises woven piezoelectric fibers, pneumatic tubing,and/or hydraulic tubing.
 4. The system recited in claim 2, wherein thedata instructs the actuator to cause the mesh to transition from (a)being in the malleable state to being in the rigid state, or (b) beingin the rigid state to being in the malleable state.
 5. The systemrecited in claim 2, wherein, in the rigid state, the mesh is flat. 6.The system recited in claim 2, wherein the immersive environment is anaugmented-reality, virtual-reality, enhanced-reality, orimmersive-reality environment.
 7. The system recited in claim 2, whereinthe immersive environment comprises a virtual peripheral or a surgicalinstrument.
 8. The system recited in claim 7, wherein the virtualperipheral comprises a keyboard, a menu, or a mouse, or the surgicalinstrument comprises a scalpel.
 9. The system recited in claim 2,wherein the wearable device is situated in a body suit, a sleeve, aglove, or footwear.
 10. The system recited in claim 2, wherein thewearable device is configured to be attached to a hand, a foot, an arm,a leg, a head, or a neck.
 11. The system recited in claim 2, wherein theactuator is a hydraulic actuator, a pneumatic actuator, an electricactuator, a thermal actuator, a magnetic actuator, or a mechanicalactuator.
 12. The system recited in claim 2, wherein the actuatorcomprises a piezoelectric actuator, a piezoceramic actuator, adielectric elastomer actuator, a polyvinylidene fluoride actuator, anelectrostatic actuator, a microelectromechanical (MEMS) actuator, or amagnetorheological actuator.
 13. The system recited in claim 2, wherein:the actuator is configured to restrict movement of a body part of awearer of the wearable device based at least in part on the data, and/orthe actuator is configured to cause movement of the body part of thewearer of the wearable device based at least in part on the data. 14.The system recited in claim 2, wherein: the actuator is configured toemulate a sensation based at least in part on the data.
 15. The systemrecited in claim 2, further comprising: a sensor capable of beingcoupled to the processor.
 16. The system recited in claim 15, whereinthe data is first data, and wherein, when executed by the processor, themachine-executable instructions further cause the processor to: obtainsecond data from the sensor, the second data indicating the position ororientation of the at least a portion of the wearable device.
 17. Thesystem recited in claim 16, wherein: the second data further indicates adetected characteristic of an object in contact with the sensor, theobject being external to a wearer of the wearable device, and therendering information is based at least in part on the detectedcharacteristic.
 18. The system recited in claim 17, wherein the detectedcharacteristic comprises a texture, a resistance, a temperature, ahardness, a pressure, a density, a coefficient of friction, or aviscosity.
 19. The system recited in claim 16, wherein the data furtherindicates a detected characteristic of an object in contact with thesensor, the object being external to a wearer of the wearable device,and wherein, when executed by the processor, the machine-executableinstructions further cause the processor to: obtain, from memory,information representing the detected characteristic at a prior time atwhich the object was previously in contact with the sensor, and identifya change in the object between the prior time and a present time. 20.The system recited in claim 19, wherein the object is a body part of apatient.
 21. The system recited in claim 15, wherein the sensor is agyroscopic sensor or an acceleration-detecting sensor.